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EXECUTIVE  SUMMARY 


This  guide  has  been  prepared  for  use  by  Department  of  Defense  (DoD)  and  Defense 
Contractor  Program  and  Engineering  Managers.  It  is  intended  to  provide  guidance  for 
the  implementation  of  a  process  for  reliability  assurance  that  incorporates  the 
principles  and  practices  of  quality  management.  Paragraphs  2  through  6  herein 
provide,  for  each  DoD  Acquisition  Phase,  a  time  phased  description  of  the  activities, 
their  input  and  output  products,  and  tracking  metrics  recommended  for  a  quality 
management  based  approach  to  reliability  assurance  and  control.  DoD  managers  can 
use  the  information  for  the  construction  of  Requests  for  Proposal,  Instructions  to 
Offerors,  Statements  of  Work,  and  Evaluation  Criteria  for  electronic  systems 
procurement.  DoD  contractors  can  use  the  information  for  subcontract  documents  and 
the  development  of  internal  processes  that  implement  the  contents  of  the  handbook. 
Both  DoD  and  contractor  management  can  use  the  handbook  contents,  specifically 
including  the  recommended  metrics,  for  the  development  of  design  review  and 
systems  engineering  event  criteria  in  order  to  track  and  control  programs. 

Total  Quality  Management  (TQM)  is  an  approach  to  management  that  embraces  two 
basic  principles:  1)  customer  satisfaction,  and  2)  continuous  improvement.  These 
principles  drive  several  primary  operating  characteristics.  Customer  satisfaction 
dictates  that  customer  needs  be  identified  and  translated  into  product  design  and 
manufacturing  requirements  in  a  systematic  and  comprehensive  manner.  Once  these 
requirements  have  been  identified,  they  should  be  implemented  in  a  manner  that  is 
directed  at  achieving  nearly  defect  free  products.  The  principle  of  continuous 
improvement  requires  that  design  and  manufacturing  processes  be  clearly  defined 
and  understood.  This  provides  the  framework  for  both  the  execution  of  these 
processes  by  trained  and  empowered  employees  and  the  improvement  of  these 
processes  through  benchmarking. 

The  document  that  is  the  current  basis  for  DoD  electronic  product  reliability  programs 
is  MIL-STD-785,  "Reliability  Program  for  Systems  and  Equipment  Development  and 
Production."  While  this  document  contains  many  essential  and  value-added  tasks, 
current  implementation  practices  and  insufficient  coverage  in  some  key  areas  limit  the 
effectiveness  of  the  programs  guided  by  this  document. 

The  reliability  process,  described  in  this  guide  for  each  DoD  acquisition  phase, 
combines  the  primary  operating  characteristics  of  quality  management  with  an 
emphasis  on  the  engineering  and  manufacturing  analysis  and  development  tasks 
required  for  the  prevention  of  product  defects.  Essential,  value-added  tasks  such  as 
planning,  supplier  control,  development  testing,  stress  screening  and  Failure 
Reporting  Analysis  and  Corrective  Action  have  been  retained  and  enhanced  in  the 
reliability  process  descriptions.  Particular  attention  has  been  paid  to  defining  the 
reliability  process  for  the  critical  early  acquisition  phases  that  precede  Engineering 
and  Manufacturing  Development  (EMD). 
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Traditional  reliability  programs  have  concentrated  on  minimizing  and  then  predicting 
tiie  rate  of  occurrence  of  "inherent”  failures.  The  fundamental  problem  with 
“tradtijonaT  reliability  programs  is  the  narrow  definition  of  reliability.  Product  defects 
are  presently  treated  by  various  functional  specialties  without  integration  and  with 
varying  degrees  of  attention.  There  has  also  been  inadequate  emphasis  on  the 
reliable  performance  of  seif-test  functions  and  the  variability  of  parts  and 
manufacturing  processes.  The  tools  are  available  to  correct  these  problems.  What  is 
required  is  the  insistence  of  DoD  customers  and  contractor  management  on  a 
multidiscipline  attack  on  all  defect  sources  using  the  principles  and  tasks  outlined  in 
this  handbook.  While  there  is  a  significant  gap  between  current  performance  and  the 
state  envisioned  by  this  handbook,  the  cultural  changes  needed  for  changing  the 
traditional  reliability  process  are  already  taking  place.  This  handbook  provides  a  focus 
and  sense  of  direction  for  those  changes.  The  reliability  processes  in  this  guide  retain 
requirements  for  quantitative  predictions  but  expand  the  focus  of  both  prediction  and 
prevention  to  include  defects  arising  from  all  sources.  This  approach  requires  the 
cooperative  and  concurrent  efforts  of  a  wide  variety  of  functional  specialties. 
Consequently,  the  processes  of  this  guide  must  be  implemented  by  the  actions  of  a 
multidiscipline  team,  under  the  leadership  of  a  process  owner. 

The  traditional  reliability  program  task  of  Electronic  Parts/Circuits  Tolerance  Analyses 
deals  with  examining  the  effects  of  parameter  variability.  This  task  has  been 
necessarily  expanded  in  the  processes  of  this  guide  to  encompass  variability  control, 
both  in  design  and  manufacturing.  The  Motorola  Six  Sigma  concept  has  been 
adopted  as  a  benchmark  for  this  variability  control. 

Fundamental  TQM  concepts  such  as  Quality  Function  Deployment,  Benchmarking, 
Training,  Software  Capability  Maturity  Model,  and  Quality  Evaluations,  based  on 
Malcolm  Baldrige  Award  criteria,  have  been  included  in  the  reliability  process.  These 
are  the  core  elements  of  Quality  Management  that  are  essential  to  both  customer 
satisfaction  and  continuous  improvement. 

This  guide  is  comprised  of  two  volumes.  Volume  One  defines  the  details  of  the 
reliability  process.  Volume  Two  contains  Certification  and  Audit  Procedures  for  an 
enterprise  wide  quality  assessment  patterned  after  the  Malcolm  Baldrige  Award 
criteria. 

Chapter  1  of  Volume  One  provides  an  overview  of  the  revised  reliability  phases.  It 
summarizes  the  basic  intent  of  the  new  process,  describes  the  preferred  metrics  for 
process  control,  and  provides  recommendations  for  ownership  of  the  elements  of  the 
new  process. 

The  process  descriptions  contained  in  Chapters  2  through  6  are  based  on  identifying 
critical  activities,  for  each  acquisition  phase,  along  with  their  inputs  and  outputs.  Detail 
descriptions  of  the  activity's  purpose  and  the  characteristics  of  each  input  and  output 
are  included  as  part  of  the  process  description.  In  addition  to  simplifying  the  process 
description,  the  technique  provides  a  focus  on  the  issue  of  customers  and  suppliers 
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and  allows  an  evaluation  of  the  quality  of  the  entire  process.  The  level  of  detail  of  the 
process  descriptions  is  sufficient  to  provide  an  understanding  of  what  is  required  to 
create  documents  such  as  statements  of  work.  At  the  conclusion  of  each  acquisition 
phase  process  chapter  is  a  description  of  appropriate  control  and  audit  metrics. 

Chapter  7  provides  a  case  history  example  of  the  application  of  the  revised  reliability 
process. 

Chapter  8  describes  the  application  of  the  process  principles,  defined  in  Chapters  2 
through  6,  to  the  development  of  defect  free  software. 

Implementation  of  the  processes  defined  in  this  guide  cannot,  in  general,  be 
accomplished  by  referencing  existing  specifications,  standards  or  data  items.  The 
significant  departure  from  current  practice  described  herein,  requires  careful  and 
individual  consideration  for  application  to  specific  programs. 
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1.0  INTRODUCTION 

Volume  I  of  this  handbook  defines  major  revisions  to 
the  traditional  reliability  process.  The  handbook  is 
divided  into  seven  chapters.  Chapter  1  provides  an 
overview  of  the  revised  process,  key  elements  of  the 
process  and  implementation  guidances. 

Chapters  2  through  6  provide  detail  definition  of 
tasks  to  be  performed  for  each  of  the  DoD 
acquisition  phases  plus  the  Pre-Concept  Exploration 
Phase  (pre  Milestone  0).  Chapter  7  provides  a  fact- 
based  implementation  example. 

The  handbook  provides  the  detail  necessary  to 
create  a  comprehensive  multidiscipline  approach  to 
reliability  assurance  and  control.  It  incorporates  and 
expands  the  principles  of  MIL-STD-785  and 
includes  some  fundamental  practices  required  for 
continuous  improvement.  A  salient  feature  of  the 
handbook  is  it's  expansion  of  the  term  "Reliability''  to 
encompass  product  defects  arising  from  all  sources 
(e.g.,  parts  and  design  defects,  excess  stress, 
fatigue,  drift,  manufacturing  and  assembly  defects, 
and  part  and  manufacturing  variability). 
Consequently,  the  handbook  describes  some 
essential  tasks  that  may  exceed  the  traditional 
practices  of  Reliability  specialists.  In  these  cases, 
the  Reliability  specialist  should  provide  a 
leadership/training  role  to  ensure  accomplishment  of 
the  described  task. 

This  handbook  deals  exclusively  with  issues 
affecting  electronic  product  reliability,  with  the  term 
reliability  used  in  the  broad  sense  noted  above. 
Wherever  terms  are  used  in  the  handbook  that  could 
infer  broader  issues,  it  is  intended  that  these  terms 
apply  to  r%  lability  issues.  For  example,  the  term 
Technical  Objectives”  applies  to  those  objectives 
which  relate  to  product  reliability. 

1.1  PROCESS  OVERVIEW 

Figure  1-1  provides  a  generic  description  of  the 
revised  reliability  process.  The  process  is  driven  by 
both  a  commitment  to  quality  operations  and 
customer  requirements.  The  activities  of  planning, 
design  and  analysis,  and  test  convert  the  customer 
requirements  into  a  product. 
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Figure  1-1.  Process  Overview 

The  dual  feedback  elements  of  design  review  and 
auditing  provide  control  of  the  process  and 
continuous  comparison  of  results  to  customer 
requirements.  This  generic  process  has  been 
applied  to  the  DoD  Acquisition  Phases  by 
constructing,  for  each  phase,  a  series  of  critical 
activities,  with  each  of  these  activities  having 
required  input  and  output  products,  and  providing 
auditing  requirements,  including  metrics  with 
quantitative  objectives.  Every  activity  is  defined  in 
terms  of  its  purpose,  objectives,  and  requirements. 

All  inputs  and  outputs  for  each  activity  are 
described  in  sufficient  detail  to  permit  execution. 

The  detail  covers  such  issues  as  the  technical 
content  of  the  input  or  output  product,  the  specialty 
or  function  providing/receiving  the  products,  the 
purpose  of  the  product,  requirements  for  determining 
the  acceptability  of  the  products  and  their  intended 
uses.  The  detail  descriptions  of  the  activities,  inputs 
and  outputs,  and  auditing  methods  *  a  each  DoD 
Acquisition  Phase  are  provided  in  Cnapters  2 
through  6  in  this  handbook.  Each  of  these  chapters 
also  contain  graphics  depicting  the  linkage  of  ti  e 
phase  activities,  the  time  phasing  of  the  activities 
within  a  phase,  and  all  inputs  to  and  outputs  from 
each  activity. 

The  material  in  Chapters  2  througii  6  defines  what  is  CRITICAL 
required  for  the  implementation  of  a  TQM  approach 
to  reliability  assurance  control.  The  process 
descriptions  define  both  the  critical  activities 
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directed  at  a  comprehensive  approach  to  defect 
elimination  and  the  activities  required  to  promote 
customer  involvement  and  continuous  improvement. 
These  latter  items  include  requirements  for  a  Quality 
Evaluation  in  accordance  with  Malcolm  Baldrige 
Award  criteria,  explicit  interpretation  and 
deployment  of  customer  needs,  benchmarking 
plans,  training  plans,  and  design  reviews  that  are 
focused  on  customer  requirements. 

Recommendations  for  further  customer  involvement 
are  also  contained  within  the  text  descriptions  of  the 
process  items.  These  noted  items,  when  combined 
with  dear  and  comprehensive  descriptions  of 
existing  internal  processes,  form  a  minimum 
framework  for  continuous  improvement.  The 
remaining  items  to  the  process  descriptions  are  the 
critical  technical  data,  analysis,  and  tests,  directed 
at  defect  preventiorVelimination,  and  produced 
primarily  by  the  functional  specialties  of  Reliability, 
Design,  Diagnostics  Design,  and  Manufacturing. 

The  process  definitions  of  Chapters  2  through  6 
cover  the  entire  range  of  DoD  electronic  products  in 
terms  of  both  complexity,  technical  maturity,  and 
application  environments.  Clearly,  tailoring  is  a 
necessity.  For  example,  some  equipment 
development  will  begin  in  EMD  as  a  result  of  low 
complexity  or  technical  maturity.  In  this  case, 

Concept  Exploration  and  Demonstration  Validation 
process  descriptions  should  be  reviewed  for 
applicability  of  selected  Activities,  Inputs,  or 
Outputs.  Similarly,  if  application  environments  are 
relatively  benign,  several  of  the  detailed  stress  and 
fatigue  analyses  will  not  be  applicable. 

It  is  not  necessary,  nor  is  it  expected,  that  all  of  the 
Activities/Inputs/Outputs  will  be  uniquely  created  for 
every  program.  Maximum  use  of  existing  information 
must  be  accomplished.  Furthermore,  those 
suppliers  who  are  most  aware  of  customer  needs 
and  problems  and  most  committed  to  the  practice  of 
TQM  principles  will  have  the  easiest  time  of 
implementing  the  processes  of  Chapters  2  through  6. 

1.2  CONTROL  AND  AUDIT 

Each  acquisition  phase  process  description  defines 
a  procedure  for  auditing  and  control.  The  auditing 
methods  are  tailored  to  both  the  objectives  of  each 
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DoD  acquisition  phase  and  the  level  of  detail  design 
and  manufacturing  information  available  during 
these  phases.  The  procedures  are  directed  at 
promoting  customer/supplier  involvement, 
especially  during  the  early  acquisition  phases 
where  most  of  the  product's  reliability  attributes  are 
established.  Similar  to  the  process  itself,  the  audit 
procedures  require  the  participation  of  multiple 
disciplines  whose  focus  is  on  the  integration  of  tasks 
essential  to  defect  prevention  and  elimination. 

Traditional  approaches  to  the  audit  function  have 
involved  review  and  approval  of  specifically 
designated  data  items.  The  data  are  typically 
requested  by  diverse  functional  groups.  This 
approach  is  generally  late  in  terms  of  its  capability 
for  influencing  the  process.  Evaluation  of  data  is 
entirely  focused  on  an  output  product  and  often 
does  not  require  an  evaluation  of  the  quality  of 
required  inputs.  Finally,  it  does  not  ensure  that  data 
affecting  reliability  but  developed  by  different 
functional  specialties  is  integrated  with  the  review  of 
reliability  data. 

The  audit  procedures  described  herein  consist  of 
five  elements  tailored  to  the  program  phase.  These 
are:  1 )  Risk  Assessment,  2)  Defect  Rate,  3)  Six 
Sigma  Design,  4)  Six  Sigma  Manufacturing,  and  5) 
Product  Performance  Assessment.  Figure  1-2 
shows  the  application  of  these  elements  in  the 
contracted  DoD  Acquisition  Phases.  Each  of  the 
elements  provides  a  trackabie  measure  of  merit  for 
controlling  the  process  within  a  phase.  As  shown  in 
the  figure,  more  than  one  metric  is  required  to  assess 
the  status  of  the  reliability  process.  The  metrics 
support  each  other  and  provide  a  comprehensive 
assessment  of  the  process  status. 
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Figure  1-2.  Tailored  Auditing  Elements 


The  key  to  the  successful  implementation  of  the 
control  procedures  defined  herein  is  maximum 
customer/supplier  involvement.  Most  of  the  auditing 
elements  are  not  subject  to  rigid  rules  or 
specifications.  The  audit  elements  provide 
quantitative  results  which  must  be  interpreted  within 
the  context  of  the  overall  quality  of  the  process.  The 
customer  is  responsible  for  assessing  the  credibility 
of  the  quantitative  results. 

1.3  IMPLEMENTING  THE  PROCESS  RESPONSIBILITY  FOR 

THE  PROCESS 

Figure  1-3  (pages  1-8  through  1-16)  presents  a 
suggested  Customer/Supplier  matrix  which  provides 
an  overview  of  the  customers  and  suppliers  for  each 
Reliability  Process  input  or  output  identified  within 
the  roadmaps  provided  in  Chapters  2  through  6. 

The  Block  Number  in  the  far  left  column  identifies  the 
acquisition  phase  block  number  within  each 
roadmap.  The  first  digit  of  the  block  number  defines 
the  specific  phase  of  the  acquisition  process;  e.g.,  2 
is  the  Pre-Concept  Exploration  Phase,  3  is  the 
Concept  Exploration  and  Definition  Phase,  etc. 
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The  Input/Output  (I/O)  column  identifies  the  item  as 
an  input  (from  a  supplier)  or  an  output  (to  a 
customer). 

The  columns  to  the  right  of  the  VO  column  identifies 
the  customers  and  suppliers  for  each  input  or  output 
Asm. 

Figure  1-4  identifies  the  major  functional  area 
customers  and  suppliers  utilized  within  the  matrix 
and  typical  subdivisions  of  these  areas. 


•  Reliability  Engineering 

•  Design  Engineering 

-  Electrical  Design 
-Mechanical  Design 

-  Structural  Design 

-  Diagnostics/Testability  Design 

-  Support  Equipment  Design 

•  Engineering  Technologies 
-Thermodynamics 

-  Structural  Dynamics 

-  Strength  Engineering 

-  Components  Engineering 

-  Operations  Analysis 

-  Maintainability  Engineering 

-  Materials  and  Process 
Engineering 

-  System  Safety  Engineering 


•  Logistics  Support  Function 

-  Logistic  Support  Analysis 

-  Field  Service  Engineering 

•  Manufacturing 

-  Manufacturing  Engineering 

-  Product  Repair 

-  Producibil'rty 

-  Packaging  Engineering 

•  Support  Groups 

-  Company  and  Functional 
Department  Library/Archives 

-  Contract  Administration 

-  Engineering  Estimating 

-  Planning  and  Scheduling 
-Marketing 

-  Procurement 

-  Data  Processing 


•  Test  and  Evaluation  Engineering 

•  Subcontractors/Suppliers 

•  Program  Management 

•  External  Customer  (e.g.,  DoD, 
Prime  Contractor) 
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Figure  1-4.  Major  Functional  Areas  and  Subdivisions 

In  the  cases  where  either  a  multiple  supplier  is 
shown  for  the  output  of  an  activity,  or  the  output 
exceeds  that  traditionally  provided  by  Reliability 
specialists,  the  multiple  suppliers  must  function  as  a 
multidiscipline  team  under  the  suggested  leadership 
of  the  Reliability  Process  owner. 

The  material  presented  in  Chapters  2  through  6  can 
be  assembled  into  process  documentation  for 
internal  use,  applied  in  Statements  of  Work  or 
Requests  for  Proposal  for  contracted  work,  or  used 
for  the  establishment  of  design/program  review  entry 
and/or  exit  criteria. 


The  primary  focus  of  continuous  improvement  is  DATA  ITEMS 

customer  satisfaction.  This  implies  a  customer 
responsibility  to  accurately  and  completely  identify 
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his  needs.  In  order  to  foster  this  responsibility, 
unique  data  item  descriptions  and  requirements, 
specific  to  each  contract,  are  recommended  over 
the  current  “cookbook”  approach.  Data  items  may 
be  identified  based  on  Figure  1  -3  for  those  items 
where  an  external  customer  is  a  “customer”  of  input 
or  output  data.  The  descriptions  of  these  items 
contained  in  the  paragraphs  shown  in  Figure  1  -3 
should  be  used  as  the  basis  for  creating  tailored 
Contract  Data  Requirements  Lists  and  Data  Item 
Descriptions. 
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Figure  1-3.  Customers  and  Suppliers  Matrix 
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Figure  1-3  (Continued).  Customers  and  Suppliers  Matrix 
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2.0  INTRODUCTION 

This  chapter  defines  the  Activities  related  to  the 
development  of  reliability  attributes  that  should  take 
place  during  the  Pre-Concept  Exploration  phase. 
This  phase  provides  the  necessary  basis  for  all 
succeeding  phases  of  development  and  acquisition. 
Figure  2-1  identifies  the  Activities  and  provides  a 
general  time  line  for  the  sequencing  of  those 
Activities.  In  any  organization,  there  will  usually  be 
several  Pre-Concept  Exploration  programs  taking 
place  simultaneously.  These  programs  will  be  a 
mixture  of  contracted  R&D  and  independent  R&D. 

All  of  these  programs  should  be  conducted  in 
accordance  with  the  principles  described  in 
paragraphs  2.1.1  through  2.1.6.  When  the  research 
is  contracted  R&D,  the  customer  may  wish  to 
incorporate  tailored  versions  of  the  Activities, 

Inputs,  and  Outputs  into  technical  documentation, 
such  as  Specifications,  Data  Item  Descriptions,  and 
Statements  of  Work.  The  customer  should  also  look 
for  evidence  that  the  principles  of  paragraphs  2.1 .1 
through  2.1.6  are  being  applied  to  other  related,  but 
noncontract,  R&D.  The  implementation  of  several  of 
these  activities  may  be  governed  by  a  single  set  of 
organization  wide  rules  or  practices.  The 
procedures  governing  design  reviews  are  an 
example  of  such  organization-wide  practices.  In  all 
cases,  the  inputs  and  outputs  of  the  activities  of 
programs  being  conducted  in  parallel  must  be 
coordinated  to  avoid  duplication  and  maintain  a 
consistent  focus  on  customer  satisfaction. 

2.1  SUMMARY  OF  ACTIVITIES 

The  key  focus  of  all  Pre-Concept  Exploration  phase 
activity  is  the  development  of  affordable  and 
militarily  useful  technology.  The  Activities,  Inputs, 
and  Outputs  of  paragraphs  2.1.1  through  2.1.6 
support  this  focus  by  emphasizing  the  definition  of 
customer  requirements,  the  development  of  defect 
reduction  design  and  manufacturing  techniques, 
and  the  creation  of  risk  reduction  plans.  The  phase 
activities  also  emphasize  the  development  of  plans 
and  products  that  affect  the  quality  of  the  work 
performed  in  all  subsequent  acquisition  phases.  The 
six  phase  Activities  are: 


APPLICATION  OF 
ACTIVITIES,  INPUTS, 
AND  OUTPUTS 
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(1)  Interpret  Customer  Needs 

(2)  Program  Planning 

(3)  Assess  Technologies  and  identify  Risks 

(4)  Tradeoff  Analysis 

(5)  Concept  Development 

(6)  Requirements  and  Design  Reviews 

The  most  critical  of  these  activities  is  the 
interpretation  of  customer  needs  and  the  translation 
of  these  needs  into  technical  objectives.  Both  the 
customer  and  supplier  must  focus  on  the  accurate, 
systematic,  and  documented  definition  of  customer 
needs.  Active  customer  participation  in  this  activity 
is  a  requirement  for  success. 

Program  Planning  should  be  a  natural  result  of 
broadly  applicable,  organization-wide  process 
descriptions.  In  a  high  quality  organization  it  should 
be  expected  that  existing  process  descriptions 
would  address  most  of  the  Activities,  Inputs,  and 
Outputs  described  in  paragraphs  2.1.1  through 
2.1.6. 

Assessing  Technologies  and  Identifying  Risks  is  the 
second  critical  activity  of  the  Pre-Concept 
Exploration  phase.  All  technology  development 
requires  the  concurrent  characterization  of  reliability 
attributes,  including  the  definition  of  manufacturing 
technology  for  the  purpose  of  initiating  process 
control  techniques.  This  characterization  must  focus 
on  defect  prevention  and  plans  for  maturing  the 
reliability  attributes. 

Trade  Studies  are  the  vehicle  for  achieving 
balanced  designs.  They  should  be  based  on 
interactions  among  technical  objectives,  and  the 
weights  assigned  to  the  measures  of  merit  used  to 
evaluate  the  goodness  of  the  design  alternatives 
must  be  traceable  to  customer  defined  priorities. 

Concept  Development  synthesizes  the  results  of  the 
above  activities  into  recommended  systems 
implementations. 

Requirements  and  Design  Reviews  bring  discipline 
and  a  focus  on  achieving  objectives  into  the 
technology  development  process.  These  are  the 
vehicles  for  maintaining  a  focus  on  the  satisfaction 
of  customer  needs. 


DEFINE  THE  NEEDS 
OF  THE  CUSTOMER 


DEFECT  PREVENTION 


CUSTOMER 

SATISFACTION 
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This  phase  does  not  contain  a  requirement  for  a 
formal  quality  evaluation  activity.  World  Class 
organizations  will  have  already  defined  the  quality 
level  of  their  operations  and  have  adopted  plans  for 
continuous  improvement  For  organizations  that 
have  not  performed  these  tasks,  the  Program 
Planning  activity  includes  requirements  for 
implementing  internal  evaluations.  If,  in  the  view  of 
the  customer,  a  quality  evaluation  is  required  for  a 
specific  contract,  the  Quality  Evaluation  Activity  of 
paragraph  3.1 .1  should  be  implemented. 


2.1.1  ACTIVITY  -  INTERPRET  CUSTOMER  QUALITY  FUNCTION 

NEEDS  DEPLOYMENT 

A  systematic,  comprehensive  process  for  defining 
customer  needs,  the  priorities  associated  with  these 
needs,  and  the  translation  of  these  needs  into 
quantified  technology  objectives  must  be 
implemented  for  this  phase.  Quality  Function 
Deployment  is  the  recommended  tool  for 
implementing  this  activity.  This  technique  should  be 
applied  to  both  contracted  and  independent  R&D. 

There  should  be  evidence  of  a  coordinated 
approach  to  assembling  definitions  of  customer 
needs  from  a  variety  of  sources,  such  as  marketing, 
customer  visits,  and  existing  programs.  Customer 
obligations  for  this  activity  consist  of  completely 
defining  all  needs  and  objectives  for  contracted 
technology  research. 
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2.1 .1.1  INPUT  -  CUSTOMER  NEEDS  AND 
OBJECTIVES 

The  supplier  must  have  a  program  for  broad  and 
frequent  direct  customer  contact  to  supplement 
documented  expressions  of  customer  needs,  such 
as  Air  Force  Technical  Area  Plans,  Technology 
Transition  Demonstrators,  and  Navy  Broad  Agency 
Announcements,  and,  for  contracted  research, 
Mission  Descriptions,  Statements  of  Work,  and 
Technical  Specifications.  All  personnel  who  will 
have  customer  contact  must  be  made  aware  of  the 
need  to  identify  the  comprehensive  range  of 
customer  needs  and  priorities,  inclusive  of  reliability 
attributes  and  manufacturing  issues.  The  supplier 
must  also  have  a  method  for  transmitting  customer 
needs  information  across  all  research  programs. 

2.1 .1.2  INPUT  -  EXPERIENCE  DATA 

The  supplier  must  ensure  that  planned  technology 
research  is  subject  to  review  by  experts  in  all  areas 
so  that  all  customer  needs  are  identified  and 
properly  translated  into  technical  objectives.  The 
experience  data  base  should  include  u  thorough 
and  documented  understanding  of  both  good  and 
bad  customer  experience  with  current  products  as  a 
preferred  method  for  aiding  in  the  identification  of 
needs  and  constraints. 

2.1 .1.3  OUTPUT  -  TECHNICAL 
OBJECTIVES 

The  supplier  provides  a  definition  of  specific 
technical  attributes  and  parameters  that  will  satisfy 
all  customer  needs.  Quality  Function  Deployment  is 
the  recommended  method  for  defining  and 
documenting  technical  objectives.  This  definition 
includes  quantitative  objectives  for  each  of  the 
attributes  and  parameters.  The  customer  needs  must 
be  given  weighting  factors  which  represent  priority 
levels.  These  factors  will  be  used  to  define 
weighting  factors  for  each  of  the  technical  attributes 
and  parameters.  At  this  early  stage  of  development, 
it  is  important  that  quantified  attributes  and 
parameters  be  treated  as  objectives  rather  than 
requirements. 

2.1.1 .4  OUTPUT  -  RESEARCH 
REQUIREMENTS 

The  supplier,  in  concert  with  the  customer,  must,  for 
the  technology  or  system  being  developed,  list  and 
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describe  all  related  research  in  order  to  avoid 
duplication  and  maximize  resources.  As  the 
research  effort  proceeds,  additional  research 
necessary  to  enable  the  developing  technology 
must  be  identified  for  inclusion  in  future  plans.  ** 

These  lists  of  related  and  enabling  research  are  to 
be  maintained  by  the  suppliers. 

2.1. 1.5  OUTPUT  -  TRADE  STUDY 
CANDIDATES 

The  supplier  must  identify  candidates  for  trade 
studies  to  be  performed  during  the  research 
program.  The  Quality  Function  Deployment 
technique  can  assist  in  defining  these  candidates  by 
evaluating  the  interactions  among  technical 
objectives.  Trade  Off  Analyses  are  used  to  optimize 
a  technical  attribute  or  parameter  that  interacts  with 
other  technical  attributes  or  parameters.  For 
example,  radar  detection  range  is  a  function  of 
power,  weight,  volume,  receiver  sensitivity, 
reliability,  and  cost.  These  analyses  are  also  used 
to  select  between  alternative  technologies  or  system 
concepts. 

EXAMPLE  OF  INTEGRATED  RESEARCH 

Figure  2-3  illustrates  the  concept  of  cooperative 
research.  In  order  to  develop  reliable  signature 
reduction  techniques,  a  major  aircraft  company  has 
integrated  the  research  activities  of  several 
specialty  areas.  A  signature  technology  group  is 
responsible  for  developing  basic  materials  signature 
properties,  the  measurement  of  these  properties,  and 
analytic  methods  for  estimating  contributions  to 
signature  levels.  A  second  specialist  group  is 
responsible  for  research  into  long-term  aircraft 
application  of  composite  materials,  while  a  third 
specialty  group  is  responsible  for  developing 
technology  that  enhances  reliability  and  reduces 
support  resources.  Each  of  the  specialty  groups 
have  research  programs  directed  at  specific 
technology  issues.  Synergism  has  been  achieved 
by  integrating  the  research  of  these  diverse  groups 
to  emphasize  high  reliability,  low  signature  aircraft 
materials  and  design.  For  example,  aircraft  access 
doors  were  being  designed  for  optimum  signature 
performance.  These  designs  now  accommodate 
constraints  based  on  access  frequency,  durability, 
safety,  fastener  styles,  and  cure  times  for  seal 
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materials.  Research  into  these  issues  is  now  being 
accomplished  as  an  integral  part  of  design  for 
signature  reduction. 


2.1.2  ACTIVITY  -  PROGRAM  PLANNING 

Program  Planning  is  necessary  for  each  research 
program  in  order  to  define  specific  inputs  and 
outputs,  the  responsibility  for  these  inputs  and 
outputs,  the  schedule  for  these  inputs  and  outputs, 
and  the  resources  to  be  allocated,  all  based  on 
customer  priorities.  As  a  baseline,  the  plan  will 
consider  the  Activities,  Inputs,  and  Outputs 
contained  in  paragraphs  2.1.1  through  2.1.6.  If  an 
organization  has  a  set  of  clear,  comprehensive,  and 
integrated  process  descriptions,  embracing  the 
intent  of  paragraphs  2.1.1  through  2.1.6,  these 
should  be  used  in  lieu  of  separately  prepared 
program  plans.  All  involved  specialty  disciplines 
(functions)  must  concur  with  the  content  of  program 
plans. 

2.1 .2.1  INPUT  -  FUNDING  PROFILES 

The  supplier  should  have  a  funding  process  that  is 
interactive,  flexible,  and  responsive  to  customer 
needs  and  priorities.  Funding  profiles  should  be  an 
element  of  initial  requirements  reviews. 
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2.1 .2.2  INPUT  -  CONTRACT  OR  RESEARCH 
REQUIREMENTS 

This  input  applies  to  both  contracted  and 
independent  research.  It  consists  of  a  complete 
description  of  overall  program  objectives, 
schedules,  task  definitions,  and  deliverables 
(reports). 


2.1 .2.3  INPUT  -  TECHNICAL  OBJECTIVES 

Output  2.1 .1 .3  serves  as  an  input  to  the  Program 
Planning  Activity. 

2.1. 2.4  INPUT  -  TRADE  STUDY 
CANDIDATES 

Output  2.1 .1 .5  serves  as  an  input  to  the  Program 
Planning  Activity.  Additional  Trade  Study 
candidates  may  be  identified  based  on  customer 
needs  and  supplier  experience.  The  candidate  list 
should  define  specific  measures  of  merit  being 
evaluated  for  each  alternative  technology  or  design. 
Reliability  attributes  should  always  be  included  as 
trade  study  measures  of  merit. 

2.1. 2.5  OUTPUT  -  TAILORED  PROGRAM 
PLAN 

The  supplier  should  prepare  program  plans  for  all 
research  programs  using  the  Activities,  Inputs,  and 
Outputs  of  paragraphs  2.1.1  and  2.1.3  through  2.1.6 
for  guidance.  Plans  should  be  tailored  based  on  the 
complexity  and  scope  of  the  research  programs. 
Existing  organization-wide  process  descriptions 
should  be  used  if  they  are  comprehensive  and 
cover  the  intent  of  paragraphs  2.1.1  and  2.1.3 
through  2.1.6.  Plans  must  always  identify 
acceptance  criteria,  responsibility,  and  schedule  for 
both  input  and  output  products. 

2.1. 2.6  OUTPUT  -  COMPUTER  AIDED 
ENGINEERING  (CAE)  TOOLS 

The  supplier  must  continuously  maintain  and  update 
the  definition  of  hardware  and  software  design 
automation  tools  and  the  interoperability  of 
equipment  and  data  bases.  These  plans  would 
generally  be  done  on  an  organization-wide  basis 
and  should  include  plans  and  provisions  for  training. 
Concurrent  Engineering  demands  automation  and 
the  capability  for  the  interchange  of  data. 
Consideration  of  customer/use  access  to  data  bases 
should  always  be  part  of  these  plans. 
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2.1. 2.7  OUTPUT  -  TECHNICAL 
OBJECTIVES  FLOWDOWN 

The  initial  Quality  Function  Deployment  will  identify 
the  top  level  technical  attributes  and  parameters 
associated  with  satisfying  customer  needs,  the 
quantitative  objectives  for  these  attributes  and 
parameters,  and  a  quantification  of  the  relative 
importance  of  each  attribute  or  parameter  based  on 
customer  priorities.  If  required  for  further  analysis  or 
trade  studies,  these  top  level  attributes  and 
parameters  will  be  decomposed  or  allocated  to  lower 
level  parameters  and  objectives.  Traceability  to 
customer  needs  and  quantification  of  importance 
based  on  customer  priorities  will  be  maintained. 

2.1. 2.8  OUTPUT  -  CONTINUOUS 
IMPROVEMENT  PLANS 

The  supplier  must  be  developing  and  implementing 
organization-wide  plans  for  continuous 
improvement.  Evidence  should  exist  that  shows  that 
the  supplier  has  embarked  on  a  program  that 
includes  comprehensive  and  specific  process 
descriptions,  an  assessment  of  the  quality  of 
operations,  benchmarking  plans,  and  training  in 
topics  such  as  QFD,  design  for  manufacturing  and 
variability  control. 


2.1.3  ACTIVITY  ■  ASSESS  TECHNOLOGIES  DEFECT  PREVENTION 
AND  IDENTIFY  RISKS 

The  supplier  must  begin  the  process  of  developing 
methods  for  preventing  or  eliminating  defects  in 
product  reliability  attributes  (life,  durability,  defect 
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rate,  BIT  performance).  This  Activity  is  applicable  to 
all  technology  or  system  concepts  research 
programs.  The  activity  includes  identifying  initial 
design,  application,  and  derating  criteria, 
environmental  sensitivity,  critical  parameters  and 
functions,  parameter  variability,  projected 
manufacturing  technology,  and  quantitative 
performance  expectations  for  reliability  attributes. 

2.1 .3.1  INPUT  -  TECHNICAL  OBJECTIVES 

Output  2.1. 1.3  or  2.1. 2.7,  as  appropriate,  serve  as 
an  input  to  Activity  2.1.3.  This  input  identifies  the 
technical  attributes  and  parameters  and  their 
baseline  quantitative  objectives. 

2.1 .3.2  INPUT  -  TECHNOLOGY/CONCEPT 
DATA 

The  new  technology  or  systems  concepts  must  be 
completely  defined,  including  identification  of 
functionally  and/or  physically  similar  existing 
technologies  or  systems.  These  comparisons 
should  define  the  environmental  and  use 
applications  for  both  the  existing  technology  or 
systems  and  the  new  technology  or  systems. 

Factors  that  should  be  identified  for  the  new 
technology  or  systems  include  first  order  estimates 
of  environmental  and  use  application,  environmental 
constraints  or  sensitivities,  critical  attributes, 
parameters,  or  functions,  variability  range  of  critical 
parameters,  packaging  concepts,  and  projected 
manufacturing  technologies  or  processes. 

2.1 .3.3  INPUT  -  EXPERIENCE  DATA 

The  supplier  must  assemble  all  potentially  relevant 
experience  data.  This  data  includes  customer  and 
internal  Lessons  Learned,  user  reliability  attribute 
performance  data  for  functionally  or  physically 
similar  technologies/systems,  related  research 
results,  existent  design  and  application  guides  (e.g., 
derating,  design  margins),  and  current 
manufacturing  processes  defect  rate  data.  This 
input  should  be  used  as  an  opportunity  for  customer 
interaction. 

2.1 .3.4  OUTPUT  -  TECHNOLOGY 
ASSESSMENT  REPORT 

This  report  should  accomplish  two  objectives:  1) 
define,  quantitatively,  the  expected  performance 
levels  of  the  technology  or  system  reliability 
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attributes,  and  2)  provide  an  initial  definition  of 
design  constraints  and  application  guides  which  will 
prevent  defects.  Quantitative  performance 
expectations  are  to  be  based  on  comparisons  to 
user-experienced  performance  levels  for 
functionally  or  physically  comparable  current 
technology  or  systems.  These  estimates  are  to  be 
supported  by  specific  engineering  rationale  that 
include  causal  analysis  of  existing  performance 
deficiencies  in  reliability  attributes.  All  critical 
parameters  and/or  functions  should  be  identified  via 
Failure  Modes  Effects  or  similar  analyses.  Initial 
estimates  of  allowable  parameter  variability  should 
be  made.  The  report  will  indude  descriptions  of 
applicable  lessons  learned,  con'  traints  and 
application  guidelines/design  m  .gins,  definition  of 
environmental  stress  sensitivities,  and  required 
manufacturing  technologies  and  processes. 

2.1. 3.5  OUTPUT  -  RISK  ASSESSMENT  RISK  REDUCTION 

REPORT 

This  report  should  identify  areas  of  uncertainty 
relative  to  achieving  the  technical  objectives  for 
reliability  attributes,  and  the  plans  for  eliminating 
those  uncertainties.  These  risk  areas  include  critical 
parameter  variability,  environmental  stress 
sensitivities,  the  absence  of  design  and  application 
criteria,  and  the  lack  of  definition  of  manufacturing 
processes  and  technology.  Reliability  performance 
expectations  that  are  not  justified  by  engineering 
rationale  should  be  considered  a  risk  area. 

Inadequate  substantiation  of  reliability  attributes  (life, 
durability,  defect  rates,  testability  potential, 
parameter  variability,  and  manufacturing  defect 
rates)  for  new  technologies  are  always  a  risk  area. 

2.1. 3.6  OUTPUT  -  RESEARCH 
RECOMMENDATIONS 

A  supplier  should  have  procedures  in  place  that 
cause  the  results  of  current  research  to  influence 
and  guide  future  research.  Each  research  or 
development  activity  requires  a  roadmap  to  define 
further  related  research,  particularly  into  areas 
identified  as  risks  in  Output  2.1 .3.5. 
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2.1.4  ACTIVITY  -  TRADE-OFF  ANALYSIS  BALANCED  DESIGNS 

Trade-Off  analysis  is  a  formal  method  for  making 
cost-effective  decisions  during  the  development. 

Each  research  program  must  have  an  identified  list 
of  trade  studies,  the  measures  of  merit  to  be 
compared  for  alternative  technologies  or  system 
concepts,  and  a  weighting  factor  for  each  measure 
of  merit  that  is  traceable  to  customer  priorities. 

Measures  of  merit  would  include  such  things  as 
weight,  processing  speed,  sensitivity,  cost,  failure 
rate,  etc.  For  the  purpose  of  reaching  a  trade  study 
conclusion  each  of  the  measures  of  merit  are 
assigned  a  weighting  factor  representing  the  relative 
importance  of  the  measure  of  merit  to  the  customer. 

Trade  study  candidates  are  developed  during  the 
Definition  of  Customer  Need  Activity  (2.1.1).  One  or 
more  of  the  reliability  attributes  and  measures  of 
merit  representing  manufacturing  technologies  must 
be  considered  for  all  trade  studies.  The  supplier 
must  have  a  well  defined  trade-off  process  that  is 
linked  to  the  Requirements  and  Design  Review 
Activity  (2.1.6). 

2.1 .4.1  INPUT  -  TECHNICAL  OBJECTIVES 

Output  2.1 .1 .3  or  2.1 .2.7  serves  as  an  input  to  the 
Trade-Off  Analysis  Activity.  These  are  the 
quantified  technical  attributes  or  parameters  that 
satisfy  customer  needs  and  include  a  further 
quantification  that  represents  customer  priorities. 
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2.1. 4.2  INPUT  -  TRADE  STUDY 
CANDIDATES 

Output  2.1 .1 .5  serves  as  an  Input  to  the  Trade-Off 
Analysis  activity.  The  list  of  candidates  may  change 
during  the  course  of  the  program  and  should  be 
responsive  to  changes  in  customer  needs  or 
priorities. 

2.1. 4.3  INPUT  -  TECHNOLOGY/CONCEPTS 
ALTERNATIVE  DATA 

Accurate  information  regarding  the  relationships 
among  technical  attributes  and  parameters  (e.g., 
weight  versus  reliability  attributes)  and  description  of 
alternative  technologies  or  system  concepts  is  key 
to  valid  Trade-Off  conclusions.  The  claimed 
relationships  among  technical  attributes  and 
parameters  should  be  validated  by  experiment  or 
extensive  analysis.  The  data  defining  alternative 
technologies  or  systems  concepts  should 
approximate  the  depth  and  quality  described  in 
paragraph  2.1. 3.2,  TECHNOLOGY/CONCEPT 
DATA.  Weighting  factors  assigned  to  the  measures 
of  merit  used  to  compare  alternatives  should  be 
traceable  to  customer  priorities  defined  during 
Activity  2.1.1.  Specific  reliability  and  manufacturing 
attributes  are  to  be  considered  for  all  trade  studies. 

2.1. 4.4  OUTPUT  -  TRADE  STUDY  REPORTS 

These  reports  provide  a  complete  record  of  trade  off 
analysis  results.  This  includes  the  selected 
technology  or  system  concept,  quantitative  values, 
technical  objectives,  the  trade  study  alternatives, 
methods,  and  selection  criteria.  The  report  should 
describe  how  the  selection  clearly  satisfies 
customer  needs  and  priorities. 
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2.1.5  ACTIVITY  -  CONCEPT  DESIGN  SYNTHESIS 

DEVELOPMENT 

All  research  programs  should  include  a  summary 
report  which  defines  the  quantified  technical 
objectives  for  the  technology  or  systems  concept 
and  summarizes  risk  issues  requiring  further 
research  and/or  analyses. 

2.1 .5.1  INPUT  -  TECHNICAL  OBJECTIVES 

Output  2.1 .1 .3  or  2.1 .2.7  serves  as  an  input  to  the 
Activity.  These  data  define  baseline  quantified 
technical  objectives. 

2.1. 5.2  INPUT  -  TECHNOLOGY 
ASSESSMENT  REPORT 

Output  2. 1.3.4  serves  as  an  input  to  the  concept 
development  activity.  This  input  defines  the 
performance  expectations  for  reliability  attributes, 
justified  by  engineering  rationale,  and  supported  by 
design  rules. 

2.1 .5.3  INPUT  -  RISK  ASSESSMENT 
REPORT 

Output  2. 1.3.5  serves  as  an  Input  to  Activity  2.1.5. 

The  data  summarizes  the  risks  associated  with 
achieving  expected  performance  levels  for  the 
reliability  attributes  and  the  plans  for  the  control  of 
these  risks. 
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2.1. 5.4  INPUT  -  TRADE  STUDY  REPORTS 

Output  2.1. 4.4  serves  as  an  input  to  Activity  2.1 .5 

2.1. 5.5  OUTPUT  -  PREFERRED  SYSTEM 
CONCEPT  REPORT 

All  research  programs  require  a  summary  report 
defining  the  technology  applications  or  system 
concept  and  the  linkage  between  quantified 
technical  objectives  and  customer  needs.  Specific 
constraints  or  defect  prevention  conditions  are 
defined,  as  are  risk  issues  and  risk  control  plans. 
Projected  manufacturing  technologies  or  processes 
are  also  identified.  This  report  will  reflect  a  set  of 
customer  needs  and  priorities  that  have  been 
validated  during  the  Requirements  and  Design 
Review  Activity. 


2.1.6  ACTIVITY  -  REQUIREMENTS  AND  PROCESS  DISCIPLINE 

DESIGN  REVIEWS 

This  activity  must  maintain  an  essential  discipline 
throughout  the  activities  of  this  phase  and  ensure 
that  the  "voice  of  the  customer"  is  defined  and 
heeded.  The  supplier  should  have  a  documented 
design  review  procedure  that  ensures  consistency 
and  thoroughness.  The  definition  for  the  timing  of 
Requirements  and  Design  Reviews  takes  place 
during  the  Planning  Activity  (2.1.2).  The  Inputs  to 
the  reviewed  activities  should  serve  to  define  entry 
criteria  for  reviews  with  the  Outputs  providing  the 
source  of  exit  criteria. 

2.1 .6.1  INPUT  -  ACTIVITY  OUTPUTS 

The  outputs  of  Activities  2.1.1,  2.1.3,  2.1.4,  and  2.1.5 
should  be  subject  to  the  review  process.  This 
includes  both  internal  reviews  and,  in  the  case  of 
contracted  research,  customer  reviews. 
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2.1 .6.2  INPUT  -  DESIGN  REVIEW 
PROCEDURES 

The  supplier  must  have  an  established  and 
documented  procedure  for  conducting  internal 
review  of  requirements  and  design.  The  procedure 
covers  review  frequency,  entry  and  exit  criteria, 
definition  of  participants,  agenda  requirements,  and 
closure  plans  for  action  items.  Reviews  should 
always  have  an  agenda  which  defines  the  customer 
needs  to  be  reviewed  and  specific  review  entry  and 
exit  criteria.  With  customer  concurrence,  these 
review  procedures  will  serve  as  the  basis  for 
customer  reviews  for  contracted  research.  Every 
review  must  include  a  definition  of  applicable 
customer  needs  and  a  demonstration  of  satisfaction 
of  those  needs. 

2.1. 6.3  OUTPUT  -  DESIGN  REVIEW 
REPORTS 

All  design  reviews  must  have  results  documented, 
including  a  definition  of  actions,  responsibility  for 
action  closure,  criteria  for  action  closure,  closure 
dates,  and  final  disposition  of  action  items. 

2.2  CONTROL  AND  AUDIT  QUANTIFY  THE  RISK 

While  properly  conducted  Requirements  and  Design 
Reviews  can  provide  a  significant  degree  of  control, 
quantitative  metrics  that  represent  the  effectiveness 
of  the  process  is  a  preferred  complement  to  these 
reviews.  The  selected  metric  for  the  Pre-Concept 
Exploration  phase  is  Risk  Factor.  This  criteria 
provides  a  quantitative  score  that  represents  a 
combination  of  the  likelihood  that  the  reliability 
process  will  achieve  its  intended  results  and  the 
consequences  of  failing  to  achieve  intended  results. 

The  methodology  for  determining  the  Risk  Factor 
score  follows  the  principles  of  the  Defense  Science 
Management  College  Risk  Assessment  technique. 

Figure  2-9  illustrates  the  concept  and  shows  the 
significance  of  risk  factor  scores. 
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®/T\J  •  Not  Critical  to  ttre  Program 

1/1  •  Low  Probability  of  Falure  for  the  Solution  Cbosan 


®H 


•  Moderate  Importance  to  the  Program 

•  Medkim  PrababiMy  of  Failure  for  the  Solution  Chosen 


I  •  Consequence  of  Failure  Low 


>  High  Probability  of  Failure  for  the  Solution  Chosen 


/r~\  •  Extremely  Critical  to  the  Program 

v-/"  •  Low  Probability  of  Falure  for  the  Solution  Chosen 


®/CN  •  Important  to  the  Program 
•  Low  Probability  of  Falure 


i  for  the  Solution  Chosen 


I  •  Extremely  Critical  to  the  Program 


1  High  Probability  of  Failure  for  the  Solution  Chosen 


cca-om-M-o 


Figure  2-9.  Isorisk  Contours  Highlight  Critical  Uncertainties 


The  risk  factor  formula  is:  Risk  Factor  =  P(F)  +  C(F)  -  FAILURE  OF  CRITICAL 

P(F)  x  C(F).  In  this  case,  P(f>  is  the  probability  that  the  ACTIVITIES 

reliability  process  will  not  achieve  its  intended 
results  (fail).  The  consequence  of  failure  (C(f>)  is  a 
measure  of  the  importance  of  the  technology  or 
system  concept  being  developed.  The  likelihood 
that  the  process  will  fail  is  the  sum  of  the  weighted 
likelihood  that  each  critical  process  activity  will  fail. 

Weighting  is  determined  by  the  relative  importance 
of  each  Activity  within  the  acquisition  phase.  The 
applicable  formulae,  terms,  and  weight  factors  are 
shown  in  Figure  2-10  (page  2-20).  As  indicated  in 
the  figure,  the  most  important  Activities  for  the  Pre- 
Concept  Exploration  Phase  are  the  Interpretation  of 
Customer  Needs  and  Technology  Assessment 
activities.  The  latter  activity  includes  the 
identification  of  technical  risk  and  proposed 
solutions.  Scores  for  the  individual  activity  failure 
probabilities  and  the  consequences  of  failure, 
shown  in  Figure  2-10,  are  determined  by  specific 
criteria  shown  in  Figures  2-11  through  2-15.  The 
criteria  in  Figures  2-11  through  2-14  (pages  2-21 
through  2-25)  are  directly  related  to  the  quality  of  the 
inputs  and  outputs  of  Activities  2.1.1,  2.1.3,  2.1.4, 
and  2.1.6. 
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These  scoring  criteria  can  be  tailored  to  the  needs  of 
individual  research  programs. 

For  any  specific  research  program,  the  Risk  Factor 
score  for  the  reliability  process  should  not  exceed 
0.7.  Risk  reduction  plans  for  reliability  attributes  are 
required  when  this  rule  is  violated. 

Risk  factor  evaluation  can  be  implemented  as  part  of 
the  Requirements  and  Design  review  process.  For 
objectivity  reasons,  it  is  recommended  that  risk 
factor  scores  be  developed  by  parties  not  directly 
involved  in  the  research  program.  Evaluation  and 
scoring  is  a  customer  responsibility  for  contracted 
research. 

Figures  2-1 0  through  2-1 5  completely  describe  the 
evaluation  methodology. 
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Figure  2-10.  Risk  Factor:  Pre-Concept  Exploration  Phase 
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Probability 
of  Falk** 

Critters  for  Activities  2.1.1  and  2.1.6 

Customer  needs  have  been  identified  and  prioritized  in  a  systematic  manner  and  translated  into 
product  technical  objectives.  Trade  study  candidates  have  been  identified.  Design  reviews  have  dearly 
defined  exit  criteria,  customer  needs  are  reviewed  and  updated,  compliance  with  customer  needs  are 
shown.  Related  research  and  addrtional  enabling  research  identified  and  documented. 

0.3 

Customer  needs  have  been  identified  in  a  systematic  manner  and  translated  into  product  technical 
objectives.  Trade  study  candidates  have  been  identified.  Customer  needs  reviewed  and  updated  at 
design  reviews.  Compliance  with  customer  needs  are  shown.  Related  research  and  additional  enabling 
research  identified  and  documented. 

0.5 

Customer  needs  have  been  identified  and  translated  into  product  technical  objectives.  Trade  study 
candidate  list  is  incomplete.  Design  reviews  show  compliance  with  customer  needs.  Incomplete 
identification  and  documentation  of  related  research  and  additional  enabling  research. 

0.7 

Customer  needs  only  partially  identified  with  no  dear  traceability  to  product  technical  objectives.  Trade 
study  candidate  list  is  incomplete.  Design  reviews  not  thoroughly  focused  on  customer  needs.  Very 
limited  Identification  and  documentation  of  related  research  and  additional  enabling  research. 

0.9 

Limited  customer/supplier  interchange.  Customer  needs  not  validated  with  customer,  design  reviews 
do  not  specific  ally  address  customer  needs.  No  dear  integration  of  current  research  or  definition  of 
additional  enabling  research. 

cca-0t*2-10-!>|«n 

Figure  2-11.  Probability  of  Failure  f>  :  Customer  Needs  Interpretation 
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Probability 
of  Failure 

Critiera  for  Activity  2.12 

0.1 

Funding  profiles  are  influenced  by  customer  needs  and  priorities.  All  contractAechnical 
requirements  have  been  addressed.  Phase  tasks  are  tailored  to  customer  priorities.  Customer 
needs  dearly  transmitted  to  all  personnel.  Training  needs/plans,  benchmarking  plan  for  continuous 
improvement  identified,  CAE  tools  available  and  data  bases  integrated.  Allocated  technical 
objectives  "flowed  down"  and  clearly  traceable  to  customer  needs. 

0.3 

Funding  profiles  are  partially  influenced  by  customer  needs  and  priorities.  Contract/technical 
requirements  have  been  addressed.  Customer  needs  not  adequately  transmitted  to  all  personnel. 
Training  not  specifically  addressed.  No  benchmarking  plan.  CAE  tools  list  not  correlate.  Allocated 
technical  requirements  partially  traceable  to  customer  needs. 

0.5 

Funding  profiles  not  matched  to  customer  needs  and  priorities.  Contract/technical  requirements 
have  been  addressed.  Limitec  ransmittal  of  customer  needs  to  all  personnel.  Limited  CAE  tools 
defined  and  data  bases  are  not  integrated.  Allocated  technical  requirements  not  traceable  to 
customer  needs. 

0.7 

Funding  inadequate.  Contract/technical  requirements  partially  addressed.  Very  limited  transmittal  of 
customer  needs  to  all  personnel,  very  limited  CAE  tools. 

0.9 

Funding  inadequate.  Contract/technical  requirements  partially  addressed.  No  evidence  of 
transmittal  of  customer  needs  to  personnel.  Extremely  limited  use  of  CAE  tools. 

CC  23-0132-1 1-D/)«n 

Figure  2-12.  Probability  of  Failure  P  :  Program  Planning 
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Criteria  lor  Activity  2.1.3 

To  Datermlna  tha  Aggregate  Score,  Sum  tha  Incfivldual  Scot  as  and  Divide  by  SI*  (6) 


New  Technology  Definition 


Maw  tochnoiogtewdesign  concepts  ara  fully  described.  Positive  and  negative  laatures  and  gaps  In  the  knowledge  base  are 
defined.  Expected  operating  environment  defined  and  traceable  to  customer  mission  description. 


New  tech  no!  ogles/design  concepts  are  partially  described.  Positive  and  negative  features  not  adequately  defined.  Gaps  In  the 
knowledge  base  are  partially  addressed.  Expected  operating  environment  defined  and  traceable  to  customer  mission 
description. 


New  technologies/design  concepts  are  partially  described.  Some  positive  features,  few  negative  features  defined.  Limited 
definition  of  gape  In  the  knowledge  base.  Expected  operating  environment  does  not  consider  maintenance  operations. 


Superficial  description  of  new  tedmologlesfdeslgn  concepts.  Negative  features  not  addressed.  No  description  of  gaps  In  the 
knowledge  base.  Limited  description  of  expected  operating  environment. 


New  techno) ogleVdesign  concepts  proposed  without  a  clear  assessment  of  positive  or  negative  features.  No  description  of  gaps 
In  the  knowledge  base.  Limited  description  of  expected  operating  environment 


Expected  RtUebitty  Pmtformancm 


Quantitative  performance  level  estimates  based  on  comparison  to  current  cuiwomer  experienced  levels  of  performance  of 
existing  comparable  equipment  Clear  engineering  rationale  Is  provided  for  alt  expected  Improvements.  Initial  definition  of 
desfgnTapptication  criteria  complete.  Lessons  learned  datals  available  (e  g.,  derating,  environmental  sensitivity). 


Quantitative  performence  level  estimates  based  on  comparison  to  current  levels  of  performance  of  existing  comparable 
equipment  One  or  more  key  performance  parameters  (e.g„  We,  fatigue,  false  alarm  rate,  etc.)  Is  not  Identified.  Clear 
engineering  rationale  Is  provided  for  aH  expected  Improvements.  Initial  definition  of  deslgn/appll cation  criteria  complete.  Lessons 
learned  data  Is  available  (eg.,  derating,  environmental  sensitivity). 


Quantitative  performance  level  estimates  based  on  comparison  to  current  levels  of  performance  of  existing  comparable 
equipment.  Key  performance  parameters  missing  and  engineering  rationale  for  expected  Improvements  Is  weak.  Initial  definition 
of  design/appllcalon  criteria  complete.  Lessons  teamed  data  Is  avaiable  (eg.,  derating,  environmental  sensitivity). 


Quantitative  performance  level  estimates  have  timited  reference  to  the  performance  levels  of  existing  comparable  equipment. 
Engineering  rationale  for  expected  Improvements  Is  weak.  Untiled  definition  of  design/appHcafion  criteria  or  lessons  learned. 


No  definition  of  existing  comparable  equipment  desigrVappllcation  criteria  or  lessons  learned. 


CHHcaf  Parameters  or  RincHone 


Failure  modes  effects  analysts  or  similar  analyses  have  been  conducted  to  Identity  critical  parameters  or  functions.  Fault 
detection  and  fault  tolerance  approaches  systematically  defined  and  related  to  critical  parameters  or  functions.  Variability  of 
critical  parameters  have  been  estimated. 


Failure  modes  effect*  analysis  or  similar  analyses  have  been  conducted  to  kJerSlfy  critical  parameters  or  functions.  Fault 
detection  and  fault  tolerance  approaches  systematically  defined  and  related  to  critical  parameters  or  functions.  Variability  of 
critical  parameters  not  estimated. 


Failure  modes  effects  analysts  or  similar  analyses  have  been  conducted  to  identify  critical  parameters  or  functions.  Fault 
detection  and  laud  tolerance  approaches  partially  related  to  critical  parameters  or  functions  and  not  accomplished  concurrently 
with  failure  modes  effects  analysis. 


raxure  mooes  enects  analysis  or  simiiv  analyses  nave  been  conouciea  to  loemny  critical  parameters  or 
of  linkage  between  this  analysis  and  the  definition  of  fault  detection  and  fault  tolerance  approaches. 


or  similar  analyses  not  co 
detection  or  fauit  tolerance  approaches. 


HE  TT.Iimr-.TTl'iflliTiriilFri 


Figure  2*13.  Probability  of  Failure  P(3):  Technology  Assessment 
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Criteria  lor  Activity  2.1.3 


Msnufiscfurirtg  Technology 


N«w  packaging  concept*  and  manufacturing  techndogiee  idantHted.  Sourcas  of  variability  hav*  bean  idantitad. 


Now  packaging  concepts  and  manufacturing  technologies  identified.  Variability  not  addressed. 


New  packaging  concepts  and  manufacturing  tecfinologi**  only  partially  addressed. 


New  packaging  concepts  partially  addressed.  Limited  description  of  manufacturing  technologies. 


Slight  attention  to  new  packaging  concepts  or  manufacturing  technologies. 


flfak  Reduction  Plant 


AH  risk  areas  dearly  identified  {unproven  technologies,  packaging  concepts,  or  manufacturing  technologies,  major  differences 
between  customer  needs  and  the  justified  performance  of  the  equipment).  Risk  reduction  plans  include  “measures  of  merit*  to 
be  tracked  and  tests  and  analyses  to  be  conducted. 


AH  risk  areas  dearly  identified  (unproven  technologies,  packaging  concepts,  or  manufacturing  technologies,  major  differences 
between  customer  needs  and  the  justified  performance  of  the  equipment).  Risk  reduction  plans  partially  defined,  missing  one  or 
mors  of  the  Mowing,  ‘measures  of  merit*  to  be  tracked  and  tests  or  analyses  to  be  conducted. 


Risk  areas  not  dearly  identified  (unproven  technologies,  packaging  concepts,  or  manufacturing  technologies,  major  differences 
between  customer  needs  and  the  justified  performance  of  the  equipment).  Risk  reduction  plans  partiaHy  defined,  missing  one  or 
more  of  the  Mowing:  'measures  of  merit*  to  be  tracked.  Tests  or  analyses  to  be  conducted. 


Few  risk  areas  identified  (unproven  technologies,  packaging  concepts,  or  manufacturing  technologies,  major  dHferenoes 
between  customer  needs  and  the  justified  performance  of  the  equipment).  Risk  reduction  plans  incomplete. 


No  risk  reduction  plan. 


FototbOn  tUtamrch 


Areas  lor  further  research  defined.  Proceed  in  place  lor  incorporating  these  in  company-wide  research  programa. 


Areas  for  further  research  defined.  No  specific  process  lor  incorporating  these  in  company-wade  research  programs. 


Areas  for  further  research  partially  defined.  No  specific  process  for  incorporating  these  in  companywide  research  programa 


Areas  for  further  research  partiaHy  identified.  No  evidence  of  incorporation  in  company-wide  research  programs. 


Limited  definition  of  requirements  for  further  research. 


CC  23-0132-13-O^sm 


Figure  2-13  (Continued).  Probability  of  Failure  Technology  Aeeeesment 
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Probability 
of  Failure 

Criteria  for  Activity  2.1  A 

0.1 

Trade  study  effort  comprehensive.  Candidates  selected  based  on  conflicts  in  satisfying  customer 
needs.  Alternative  technologies/de  signs  evaluated  In  a  manner  similar  to  that  done  for  the  baseline. 
Trade  study  parameter  weights  are  traceable  to  priorities  identified  during  customer  needs 
interpretation  activity. 

0.3 

Trade  study  effort  comprehensive.  Candidates  selection  not  completely  traceable  to  conflicts  in 
satisfying  customer  needs.  Alternative  technologies/designs  evaluated  in  a  manner  similar  to  that 
done  for  the  baseline.  Trade  study  parameter  weights  are  not  completely  traceable  to  priorities 
identified  during  customer  needs  interpretation  activity. 

0.5 

Trade  study  effort  limited.  Candidate  selection  not  completely  justified.  Alternative 
technologies/designs  evaluation  not  as  thorough  as  done  for  the  baseline.  Trade  study  parameter 
weights  are  not  traceable  to  priorities  identified  during  customer  needs  interpretation  activity. 

0.7 

Trade  study  effort  limited.  Candidate  selection  not  justified.  Limited  effort  in  evaluating  alternative 
technofogies/designs.  Trade  study  parameter  weights  not  related  to  customer  priorities. 

0.9 

Trade  study  effort  very  limited.  Results  not  credible. 

CCa-OItt-M-Ofm 

Figure  2-14.  Probability  of  Failure  P(4):  Tradeoff  Analyeee 


Consequence 
of  Failure 
Score 

Criteria  for  the  Consequences  of  Failure 

0.1 

Little  or  no  improvement  over  the  reliability  attributes  performance  of  current  technology  is 
needed  or  expected. 

0.3 

Some  improvement  over  the  reliability  attributes  performance  of  current  technology  is  expected 
(up  to  1 .5  times  better). 

0.5 

Significant  improvement  over  the  reliability  attributes  performance  of  current  technology  is 
expected  (from  1.5  to  3  times  better). 

0.7 

Substantial  improvement  over  the  reliability  attributes  performance  of  current  technology  is 
expected  (from  3  to  5  times  better). 

0.9 

Major  improvement  over  the  reliability  attributes  performance  of  current  technology  is  expected 
(more  than  5  times  better). 

CC2S-0132-15-O>m 

Figure  2-1 5.  Consequences  of  Failure 
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Figure  3-1.  Concept  Exploration  Phase  Activities 
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3.0  INTRODUCTION 

This  chapter  defines  the  Activities  that  should  take 
place  during  the  Concept  Exploration  Phase  to 
ensure  the  attainment  of  product  reliability  attributes. 
Figure  3-1  shows  the  essential  activities  and  a 
general  time  line  for  the  implementation  of  these 
activities.  Most  of  the  phase  activities  are  iterative: 
Trade-Off  Analyses  affect  the  assessment  of 
technologies  and  risks;  these  activities  both  affect 
the  recommendations  of  configurements  and 
requirements;  all  activities  must  be  continuously 
compared  to  customer  needs.  Detailed  descriptions 
of  the  activities  and  their  required  inputs  and  outputs 
are  contained  in  paragraphs  3.1.1  through  3.1.6. 
There  are  several  similarities  between  these 
descriptions  and  the  ones  contained  in  paragraphs 

2.1.1  through  2.1.5.  However,  while  the  activities  of 
Chapter  2  are  applied  to  a  potentially  broad  range  of 
research  programs,  a  concept  exploration  program 
is  contracted  work  that  is  focused  on  a  set  of 
customer  needs  for  fulfilling  a  specific  mission. 

3.1  SUMMARY  OF  ACTIVITIES 

The  purpose  of  the  concept  exploration  phase  is  to 
develop  and  evaluate  alternative  system  concepts 
that  satisfy  customer  needs.  A  principal  phase 
product  is  the  definition  of  a  preferred  system 
configuration  with  a  set  of  performance  objectives. 
The  development  of  the  preferred  concept  includes 
the  selection  of  high  and  medium  risk  emerging 
technologies  that  offer  solutions  to  customer  r  seds. 
A  second  primary  phase  output  is  a  risk  reduction 
plan  for  the  selected  risky  technologies.  Within  the 
phase,  analyses  and  trade-offs  are  conducted  in 
order  to  determine  system  performance  objectives 
considering  expected  environments,  use,  and 
constraints.  These  general  themes  must  be 
supported  by  the  simultaneous  development  and 
definition  of  reliability  attributes  as  defined  in 
paragraphs  3.1 .1  through  3.1 .6. 

The  Activities,  Inputs,  and  Outputs  of  the  following 
paragraphs  emphasize  the  definition  of  customer 
requirements,  the  development  of  defect  reduction 
design  and  manufacturing  techniques,  and  the 
creation  of  risk  reduction  plans  specifically  directed 
at  reliability  attributes.  The  phase  activities  also 
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demand  that  the  capability  for  quality  processes  be 
measured  and  that  steps  be  taken  to  ensure 
continuous  improvement.  The  seven  phase 
Activities  are: 

1 )  Quality  Evaluation 

2)  Interpret  Customer  Needs 

3)  Program  Planning 

4)  Assess  Technologies  and  Identify  Risks 

5)  Trade-Off  Analyses 

6)  System  Requirements  and  Configuration 
Recommendations 

7)  Requirements  and  Design  Reviews 

A  baseline  for  continuous  improvement  is 
established  by  means  of  a  pre-award  quality 
evaluation  of  bidders  for  the  concept  exploration 
contract.  This  evaluation  provides  a  score  reflecting 
the  requisite  commitment  to  total  quality  by  each  of 
the  bidders.  The  evaluation  influences  source 
selection  and  provides  a  baseline  to  measure 
necessary  improvement  during  the  concept 
exploration  and  follow-on  acquisition  phases. 

The  most  critical  of  these  above  activities  is  the 
interpretation  of  customer  needs  and  the  translation 
of  these  needs  into  technical  objectives.  Both  the 
customer  and  supplier  must  focus  on  the  accurate, 
systematic,  and  documented  definition  of  customer 
needs.  Active  customer  participation  in  this  activity 
is  a  requirement  for  success. 

Program  planning  should  be  a  natural  result  of 
broadly  applicable,  organization-wide  process 
descriptions.  In  a  high  quality  organization,  it  should 
be  expected  that  existing  process  descriptions 
would  address  most  of  the  Activities,  Inputs,  and 
Outputs  described  in  paragraphs  3.1.1  through 
3.1.7. 

Assessing  Technologies  and  Identifying  Risks  is  the 
second  critical  activity  of  the  Concept  Exploration 
phase.  All  tr-hnology  development  requires  the 
concurrent  characterization  of  reliability  attributes, 
including  the  definition  of  manufacturing  technology 
for  the  purpose  of  initiating  process  control 
techniques.  This  characterization  must  focus  on 
defect  prevention  and  plans  for  maturing  the 
reliability  attributes. 


INTERPRET  THE 
NEEDS  OF  THE 
CUSTOMER 


DEFECT  PREVENTION 
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Trade  Studies  are  the  vehicle  for  achieving 
balanced  designs.  They  should  be  based  on 
optimizing  specific  parameters  and  on  interactions 
among  technical  objectives.  The  weights  assigned 
to  the  measures  of  merit  used  to  evaluate  fie 
goodness  of  the  design  alternatives  must  be 
traceable  to  customer  defined  priorities. 

The  Systems  Requirements  and  Configuration 
Recommendations  Activity  synthesizes  the  results  of 
the  prior  activities  (excluding  the  Quality  Evaluation) 
into  a  recommended  systems  implementation  with 
quantified  objectives  for  reliability  attributes.  This 
activity  also  includes  the  development  of  a  firm  set  of 
risk  reduction  plans. 

Requirements  and  Design  Reviews  bring  discipline 
and  a  focus  on  achieving  objectives  into  the 
technology  development  process.  These  reviews 
are  the  vehicles  for  maintaining  a  focus  on  the 
satisfaction  of  customer  needs. 


BALANCED  DESIGN 


CUSTOMER 

SATISFACTION 


3.1.1  ACTIVITY  -  QUALITY  EVALUATION 

This  pre-contract  award  Activity  establishes  a 
quantitative  baseline  for  continuous  improvement.  In 
response  to  customer  instructions,  the  supplier 
conducts  an  internal  review  of  company-wide 
commitment  to  quality  practices.  This  evaluation 
uses  criteria  derived  from  Malcolm  Baldrige  Award 
criteria.  Recommended  criteria  are  provided  in 
Volume  II  of  this  guide. 
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3.1 .1.1  INPUT  -  EVALUATION 
INSTRUCTIONS 

The  customer  includes  in  his  requests  for  proposal  a 
complete  set  of  instructions  for  conducting  the 
quality  evaluation.  These  include  directions  in 
Executive  Summaries,  detail  instructions  in  the 
Instructions  to  Offerors  (Evaluation  Criteria, 

Reporting  Requirements),  and  the  relationship  of  the 
evaluation  to  source  selection  award  factors.  An 
example  of  instructions  is  provided  following 
paragraph  3.1 .1 .3.  Report  format  requirements  for  a 
Malcolm  Baldrige  Award  Application  are 
recommended. 

3.1. 1.2  INPUT  -  EVALUATION  CRITERIA 

The  customer  supplies  the  criteria  to  be  used  for  the 
quality  evaluation.  Volume  II  of  this  guide  provides 
a  set  of  recommended  criteria  patterned  after  the 
Malcolm  Baldrige  Award  criteria. 

3.1. 1.3  OUTPUT  -  QUALITY  EVALUATION 
REPORT 

The  supplier  provides  a  quality  evaluation  report  in 
accordance  with  customer  reporting  requirements. 
This  report  is  to  be  used  by  both  the  customer  and 
the  supplier.  The  customer  will  score  the  results  and 
use  this  in  source  selection.  The  supplier  should 
similarly  score  his  results  and  use  this  evaluation  to 
define  weaknesses  and  plan  the  required 
improvements. 

EXAMPLE  REQUEST  FOR  PROPOSAL 
REQUIREMENTS 

•  Sample  Executive  Summary  Language 

The  Customer  intends  to  conduct  a 
performance  risk  assessment  as  an  element  of 
the  source  selection  process.  This  assessment 
involves  the  evaluation  of  each  offeror's 
company-wide  quality  effort. 

•  Sample  RFP  Section  L  Language: 

Volume  (XX)-lnformation  for  Assessment  of 
Company-Wide  Quality  Efforts 

1.0  General.  The  Customer  intends  to 
consider  each  offeror's  company-wide  effort 


RFP  DIRECTIONS 
FOR  A  QUALITY 
EVALUATION 
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and  its  results.  Specifically,  a  performance  risk 
assessment  wil  be  conducted  to  assess  the 
effectiveness  of  the  offeror's  company-wide 
qualty  efforts. 


2.0  Specific  Information  and  Data.  The 
offeror  shaft  provide  the  information  below 
emphasizing  documented,  verifiable  evidence 
of  the  effective  implementation  of  company¬ 
wide  quality  efforts.  Actions  presently  planned 
shall  also  be  included.  Information  provided 
should  be  applicable  to  the  facilities  or  location 
where  work  under  the  proposed  contract  will 
be  performed.  The  offeror  shall  describe  the 
application  of  its  practices,  tools,  and 
techniques.  Additional  details  are  provided  in 
Section  (YY).  The  offeror's  proposal  should 
address  the  following  areas: 


Leadership  -  Describe  the  extent  to  which 
the  senior  executives  create  and  sustain  a 
clear  and  visible  quality  value  system  along 
with  a  supporting  company-wide  quality 
management  system  to  guide  all  activities  of 
the  company. 


Information  and  Analysis  -  Describe  and 
demonstrate  the  scope,  validity,  and 
management  of  data  and  information  that 
underlie  the  company's  quality  management 
system.  In  particular,  describe  how  the 
company  uses  data  to  support  a  prevention- 
based  approach  to  quality. 


Strategic  Quality  Planning  -  Describe  the 
company's  quality  priorities  and  plans  to 
achieve  them. 


Human  Resources  Utilization  -  Describe 
the  company's  practices  to  develop  and  utilize 
the  full  potential  of  tiie  work  force  and  to 
maintain  an  environment  conducive  to  full 
participation,  continuous  process 
improvement,  and  personal  and  organizational 
growth.  Summarize  quantitative  and 
qualitatively  1)  accomplishments  to  date  and 
recent  trends  in  employee  participation  in 
company-wide  quality  activity,  2)  types  of 
quality  education  and  training  provided  in 
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each  pertinent  employment  category,  and  3) 
trends  in  recognizing  employees  for 
contributions  to  the  company-wide  quality 
system. 

Quality  Assurance  of  Products  and 
Sen/ices  -  Describe  how  products  and 
services  are  continuously  improved  through 
optimization  and  improvement  of  processes.  In 
the  area  of  design  and  development,  where 
applicable,  the  description  should  include 
information  pertaining  to  the  use  of  methods, 
tools,  and  techniques  to  achieve  high  quality 
in  design.  Include  use  of  such  tools  and 
techniques  as  Concurrent  Engineering, 

Quality  Function  Deployment,  producibility 
engineering  and  planning,  design  of 
experiments,  DoD  Directive  4245.7M  - 
Transition  from  Development  to  Production, 
etc.  Include  a  description  of  how  the  offeror 
flows  the  company's  quality  focus  down  to 
subcontractor  levels. 

Results-  Provide  data  that  show  trends 
in:  a)  improvement  of  quality  of  products  and 
services  based  on  analysis  of  customer 
requirements,  analysis  of  quality  deficiency 
reports,  cycle  time  reductions,  Material 
Review  Board  actions,  scrap  and  rework,  etc., 
and  the  analysis  of  internal  business 
operations,  and  b)  improvement  in  the  quality 
of  supplies  and  services  furnished  by  other 
companies.  Provide  evidence  of  the  use  of 
results  to  overcome  and  prevent  the 
recurrence  of  problems.  Demonstrate 
application  of  the  offeror's  company-wide 
quality  activities  by  briefly  summarizing 
several  projects  that  illustrate  their  breadth  and 
effectiveness. 

Sample  RFP  Section  M  Language 

The  Customer  will  also  condr 
performance  risk  assessment  basea  on:  1) 
effectiveness  of  the  offeror’s  company-wide 
quality  activities  and  the  applicability  of  the 
offeror's  use  of  quality  practices,  tools,  and 
techniques,  and  2)  the  offeror's  past 
performance  record  demonstrated  in  terms  of 
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actual  results. 


3.1.2  ACTIVITY  -  INTERPRET  CUSTOMER 
NEEDS 

A  systematic,  comprehensive  process  for  defining 
customer  needs,  the  priorities  associated  with  these 
needs,  and  the  translation  of  these  needs  into 
quantified  technical  objectives  must  be  implemented 
for  this  phase.  Quality  Function  Deployment  is  the 
recommended  tool  for  implementing  this  Activity. 

This  is  one  of  the  most  critical  activities  of  this  phase. 
The  supplier  must  be  prepared  to  integrate  customer 
needs  defined  during  other  research  activities  with 
customer  inputs  for  this  acquisition  phase.  Customer 
contact  should  have  taken  place  prior  to  contract 
award  as  part  of  pre-RFP  release  discussions. 
Customer  needs  documented  in  contract  technical 
material  should  always  be  supplemented  and 
validated  via  direct  contact.  Customers  must 
encourage  and  support  these  contacts. 

3.1. 2.1  INPUT  -  CUSTOMER  NEEDS  AND 
OBJECTIVES 

Customer  needs  and  objectives  should  be  explicitly 
defined  in  the  contract  technical  documentation, 
such  as  Mission  Need  Statements,  Statements  of 
Work,  and  preliminary  specifications.  However,  all 
contract  documentation  must  be  reviewed  to 
descriptions  of  customer  needs.  Additionally,  the 
supplier  must  plan  for  early  direct  and  frequent 
customer  contact  to  validate  and  refine  the 
documented  needs.  These  contacts  must  focus  on  a 
multidisciplinary  definition  of  customer  needs. 


COMPREHENSIVE 
SEARCH  FOR 
CUSTOMER 
NEEDS 
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3.1. 2.2  INPUT  -  EXPERIENCE  DATA 

The  supplier  must  ensure  documented  Concept 
Exploration  needs  are  subject  to  review  by  experts 
in  all  areas  so  that  all  customer  needs  are  identified 
and  properly  translated  into  technical  objectives. 

The  supplier  must  also  ensure  that  customer  needs 
and  technical  objectives,  defined  during  prior 
research,  is  available  for  application  to  the  Concept 
Exploration  contract.  The  experience  data  base 
should  include  a  thorough  and  documented 
understanding  of  both  good  and  bad  customer 
experience  with  current  products  as  a  preferred 
method  for  aiding  in  the  identification  of  needs. 

3.1. 2.3  OUTPUT  -  TECHNICAL 
OBJECTIVES 

The  supplier  must  provide  a  definition  of  specific 
technical  attributes  and  parameters  that  will  satisfy 
all  customer  needs.  Quality  Function  Deployment  is 
the  recommended  method  for  defining  and 
documenting  Technical  objectives.  This  definition 
includes  quantitative  objectives  for  each  of  the 
attributes  and  parameters.  The  customer  needs  must 
be  given  weighting  factors  which  represent  priority 
levels.  These  factors  will  be  used  to  define  the 
relative  importance  of  each  of  the  technical 
attributes  and  parameters.  It  is  critical  that  the 
interaction  between  reliability  technical  attributes 
and  the  other  technical  attributes  be  identified.  At 
this  early  stage  of  development,  it  is  important  that 
quantified  attributes  and  parameters  be  treated  as 
objectives  rather  than  requirements. 

3.1. 2.4  OUTPUT  -  RESEARCH  RELATED  RESEARCH 

REQUIREMENTS 

The  supplier,  in  concert  with  the  customer,  must 
identify,  for  the  system  being  developed,  all  related 
research  in  order  to  avoid  duplication  and  maximize 
resources.  As  the  Concept  Exploration  phase 
proceeds,  additional  research  necessary  to  enable 
developing  technology  applicable  to  the  system 
must  be  identified  for  inclusion  in  future  plans. 

3.1. 2.5  OUTPUT  -  TRADE  STUDY 
CANDIDATES 

The  supplier  must  identify  candidates  for  trade 
studies  to  be  performed  during  the  Concept 
Exploration  phase.  The  Quality  Function 
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Deployment  technique  can  assist  in  defining  these 
candidates  by  evaluating  the  interactions  among 
technical  attributes.  Trade-off  analyses  are  used  to 
optimize  a  technical  attribute  or  parameter  that 
interacts  with  other  technical  attributes  or 
parameters.  For  example,  radar  detection  range  is  a 
function  of  power,  weight,  volume,  receiver 
sensitivity,  reliability,  and  cost.  Trade  studies  are 
also  used  to  select  between  alternative  technologies 
or  system  concepts. 


Input  Activity  Output 


CC2J01S2-1»-a>m 


Figure  3-4.  Plan  the  Program 


3.1.3  ACTIVITY  -  PROGRAM  PLANNING  ALLOCATE  THE 

Program  planning  is  necessary  for  the  Concept  RESOURCES 

Exploration  phase  in  order  to  define  specific  inputs 

and  outputs,  the  responsibility  for  these  inputs  and 

outputs,  the  schedule  for  these  inputs  and  outputs, 

and  the  resources  to  be  allocated,  all  based  on 

customer  priorities.  As  a  baseline,  planning  will 

consider  the  Activities,  Inputs,  and  Outputs 

described  in  paragraphs  3.1.3  through  3.1.7.  If  an 

organization  has  a  set  of  clear,  comprehensive,  and 

integrated  process  descriptions  embracing  the  intent 

of  paragraphs  3.1.2  through  3.1.7,  these  should  be 

used  in  lieu  of  separately  prepared  program  plans. 

Specialists  in  the  disciplines  of  Reliability,  Design, 

Testability  and  Manufacturing  must  concur  with  the 
content  of  the  program  plan. 
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3.1. 3.1  INPUT  -  FUNDING  PROFILES 

The  supplier  should  have  a  funding  process  that  is 
interactive,  flexible,  and  responsive  to  customer 
needs  and  priorities.  Funding  profiles  should  be  an 
element  of  initial  requirements  reviews  to  ensure  that 
they  are  consistent  with  customer  priorities. 

3.1. 3.2  INPUT  -  CONTRACT 
REQUIREMENTS 

This  input  consists  of  a  complete  description  of 
overall  program  objectives,  schedules,  task 
definitions,  and  deliverables  (reports). 

3.1. 3.3  INPUT  -  TECHNICAL  OBJECTIVES 

Output  3.1 .2.3  serves  as  an  input  to  the  Program 
Planning  Activity. 

3.1 .3.4  INPUT  -  TRADE  STUDY 
CANDIDATES 

Output  3.1 .2.5  serves  as  an  input  to  the  Program 
Planning  Activity.  Additional  Trade  Study 
candidates  may  be  identified  based  on  customer 
needs  and  supplier  experience.  The  candidate  list 
should  define  specific  measures  of  merit  being  used 
to  evaluate  trade  study  issues  or  alternative  designs. 
Reliability  attributes  should  always  be  included  as 
trade  study  measures  of  merit. 

3.1. 3.5  OUTPUT  -  TAILORED  PROGRAM 
PLAN 

The  supplier  should  prepare  a  program  plan  for 
Concept  Exploration  using  the  Activities,  Inputs,  and 
Outputs  of  paragraphs  3.1.2  and  3.1.4  through  3.1 .7 
for  guidance.  Plans  should  be  tailored  based  on  the 
complexity  and  scope  of  the  program.  Existing 
organization-wide  process  descriptions  should  be 
used  if  they  are  comprehensive  and  cover  the  intent 
of  paragraphs  3.1 .2  and  3.1.4  through  3.1.7.  Plans 
must  always  identify  acceptance  criteria, 
responsibility,  and  schedule  for  both  input  and 
output  products. 

3.1. 3.6  OUTPUT  -  COMPUTER  AIDED 
ENGINEERING  (CAE)  TOOLS 

The  supplier  must  continuously  maintain  and  update 
the  definition  of  hardware  and  software  design 
automation  tools  and  the  interoperability  of 
equipment  and  data  bases.  These  plans  would 


FUNDING 
RESPONSIVE  TO 
CUSTOMER  NEEDS 


CONCURRENT 

ENGINEERING 
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generally  be  done  on  an  organization-wide  basis 
and  should  include  plans  and  provisions  for  training. 
Concurrent  Engineering  demands  automaton  and 
the  capability  for  the  interchange  of  data. 

Consideration  of  customer  usa/access  to  data  bases 
should  always  be  part  of  these  plans. 

3.1. 3.7  OUTPUT  -  TECHNICAL 
OBJECTIVES  FLOWDOWN 

The  initial  Quality  Function  Deployment  will  identify 
the  system  level  technical  attributes  and  parameters 
associated  with  satisfying  customer  needs,  the 
quantitative  objectives  for  these  attributes  and 
parameters,  and  a  quantification  of  the  relative 
importance  of  each  attribute  or  parameter  based  on 
customer  priorities.  If  required  for  further  analysis  or 
trade  studies,  these  system  level  attributes  and 
parameters  will  be  decomposed  or  allocated  to  lower 
level  parameters  and  objectives.  Traceability  to 
customer  needs  and  quantification  of  importance 
based  on  customer  priorities  must  be  maintained. 

3.1. 3.8  OUTPUT  -  BENCHMARKING  AND 
TRAINING  PLANS 

The  supplier  must  develop  benchmarking  plans  to 
correct  weaknesses  identified  during  the  Quality 
Evaluation  Activity.  These  plans  should  identify  the 
specific  corrective  actions  necessary  for 
improvement  that  will  be  beneficial  to  the  conduct  of 
Concept  Exploration.  An  evaluation  of  the 
capability  of  program  personnel  to  implement  the 
Activity,  Input,  and  Output  requirements  of  Concept 
Exploration  must  be  conducted.  Training  plans  to 
correct  any  deficiencies  must  be  prepared.  Training 
plans  should  also  include  provisions  for  ensuring 
that  program  personnel  fully  understand  the  tasks  to 
be  performed  and  the  objectives  for  reliability 
attributes. 


3.1. 3.9  OUTPUT  -  SUBCONTRACTOR 
CONTROL  PLAN 

If  required,  the  supplier  must  prepare  plans  that 
describe  the  methods  used  to  flow  the  Activities, 
Input,  and  Output  tasks  of  this  phase  to 
subcontractors.  These  plans  should  define  supplier 
selection  criteria  and  the  allocation  of  technical 
objectives. 


CHAPTER  3 
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3.1.4  ACTIVITY  -  ASSESS  TECHNOLOGIES  DEFECT  PREVENTION 
AND  IDENTIFY  RISKS  ACTIVITY 

The  supplier  must  begin  the  process  of  developing 
methods  for  preventing  or  eliminating  defects  in 
product  reliability  attributes  (life,  durability,  defect 
rate,  BIT  performance).  The  activity  includes 
identifying  initial  design,  application,  and  de-rating 
criteria,  environmental  sensitivity,  critical 
parameters  and  functions,  parameter  variability, 
projected  manufacturing  technology,  and 
quantitative  performance  expectations  for  reliability 
attributes. 

As  an  inherent  part  of  this  Activity,  the  supplier  must 
establish  the  integration  of  multiple  disciplines 
focused  on  defect  prevention.  For  example,  the 
definition  of  BIT  requirements  and  fault  tolerance 
features  must  be  traceable  to  the  definition  of  critical 
functions  or  parameters  accomplished  by  Failure 
Modes  Effects  or  similar  analyses.  Similarly,  the 
initial  definition  of  key  manufacturing  processes 
should  be  focused  on  critical  functions  or 
parameters.  This  activity  addresses  both  defect 
prevention  for  new  technologies  and  defect 
elimination  for  the  existing  technology  being  applied 
to  the  system  concept. 

3.1 .4.1  INPUT  -  TECHNICAL  OBJECTIVES 

Output  3. 1.2.3  or  3.1 .3.7,  as  appropriate,  serves  as 
an  input  to  Activity  3.1 .4.  This  input  identifies  the 
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technical  attributes  and  parameters  and  their 
baseline  quantitative  objectives. 

3.1. 4.2  INPUT  -  TECHNOLOGY/CONCEPT 
DATA 

The  new  technology  and  systems  concepts  must  be 
completely  defined,  including  identification  of 
functionally  and/or  physically  similar  existing 
technologies  or  systems.  These  identifications 
should  define  the  environmental  and  use 
applications  of  the  existing  technology  or  systems. 

Factors  that  should  be  identified  for  the  new 
technology  or  systems  include  definition  of  system's 
functions  derived  from  the  mission  description, 
environmental  constraints  or  sensitivities,  critical 
attributes,  parameters,  or  functions,  variability  range 
of  critical  parameters,  packaging  concepts,  and 
projected  manufacturing  technologies  or  processes, 
especially  those  associated  with  critical  functions  or 
parameters. 

3.1. 4.3  INPUT  -  EXPERIENCE  DATA  LESSONS  LEARNED 

The  supplier  must  assemble  all  potentially  relevant 
experience  data.  These  data  include  customer  and 
internal  Lessons  Learned,  user  reliability  attribute 
performance  data  for  functionally  or  physically 
similar  technologies/systems,  results  from  related 
research,  existent  design  and  application  guides 
(e.g.,  derating,  design  margins),  and  current 
manufacturing  process  defect  rate  data.  Data 
concerning  lessons  learned  and  the  performance  of 
comparable  systems  should  be  thoroughly 
coordinated  with  the  customer. 

3.1. 4.4  INPUT  -  ENVIRONMENT  AND  USE 
DATA 

The  supplier  must  derive  first  order  environment  and 
use  data  from  the  customer-provided  Mission 
Descriptions.  These  data  should  address  the 
primary  fault-producing  external  environments,  such 
as  temperature  extremes,  temperature  cycling, 
vibration,  shock,  and  humidity  produced  by  both 
expected  operating  conditions  and  maintenance 
conditions.  Data  should  include  maximum 
conditions  and  an  estimate  of  cumulative  lifetime 
exposures  based  on  expected  missions,  basing,  and 
maintenance  frequency.  Functional  analyses  will 
be  conducted  to  define  operating  profiles  for  the 
system. 
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3.1 .4.5  OUTPUT  -  TECHNOLOGY 
ASSESSMENT  REPORT 

This  report  should  accomplish  two  objectives:  1) 
define,  quantitatively,  the  expected  performance 
levels  for  the  system  reliability  attributes,  and  2) 
provide  an  initial  definition  of  design  constraints  and 
application  guides  which  will  prevent  defects. 
Quantitative  performance  expectations  are  to  be 
based  on  comparisons  to  user  performance  levels 
for  functionally  or  physically  comparable  current 
technology  or  systems.  The  estimates  of  quantitative 
performance  for  the  proposed  system  are  to  be 
supported  by  specific  engineering  rationale  that 
include  causal  analysis  of  existing  performance 
levels.  All  critical  parameters  and/or  functions 
should  be  identified  via  Failure  Modes  Effects  or 
similar  analyses.  Initial  estimates  of  allowable 
critical  parameter  variability  should  be  made.  The 
report  will  describe  defect  reduction  elements,  such 
as  lessons  learned,  constraints  and  application 
guidelines/design  margins,  definition  of 
environmental  stress  sensitivities,  and  required 
manufacturing  technologies  or  processes  applicable 
to  both  new  and  existing  technologies  proposed  for 
use  in  the  new  system. 

This  is  a  critical  output  that  should  define  the 
following  issues: 

•  Quantitative  expectations  for  system  reliability 
attributes,  such  as  life,  durability,  BIT 
performance,  and  defect  rates  inclusive  of 
projected  manufacturing  defects.  These 
estimates  should  be  based  on  engineering 
justified  changes  in  the  conditions  which 
produced  the  user  experienced  performance  of 
comparable  equipment. 

•  Identification  of  specific  defect  prevention 
techniques,  such  as  the  incorporation  of 
lessons  learned,  design  rules  such  as 
application  guides  and  derating,  design  for 
manufacturing  principles,  and  manufacturing 
process  changes. 

•  Identification  of  the  environmental  sensitivities 
of  expected  technologies  for  the  purpose  of 
initiating  materials  characterization  testing. 


DEFECT  REDUCTION 
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•  Definition  of  critical  functions  and  parameters 
in  order  to  provide  a  basis  for  BIT  requirements 
and  fault  tolerance  features. 

•  Definition  of  the  expected  variance  in  critical 
functions  or  parameters  in  order  to  initiate  the 
process  of  variability  control  and  the 
identification  of  manufacturing  processes 
requiring  characterization  and  control. 

Generation  of  this  output  provides  an  excellent 
opportunity  for  soliciting  customer  involvement, 
particularly  in  the  definition  of  lessons  learned,  the 
performance  of  comparable  equipment,  and  in  the 
identification  of  critical  functions  or  parameters. 

3.1. 4.6  OUTPUT  -  RISK  ASSESSMENT  RISK  REDUCTION 

REPORT 

This  report  should  identify  areas  of  uncertainty 
relative  to  achieving  the  technical  objectives  for 
reliability  attributes  and  the  plans  for  eliminating 
those  uncertainties.  These  risk  areas  include  critical 
parameter  variability,  environmental  stress 
sensitivities,  the  absence  of  design  and  application 
criteria,  and  the  lack  of  definition  of  manufacturing 
processes  and  technology.  Estimates  of  reliability 
performance  that  are  not  justified  by  substantive 
engineering  rationale  should  be  considered  a  risk 
area.  Special  attention  must  be  paid  to  the 
substantiation  of  reliability  attributes  (life,  durability, 
design,  defect  rates,  BIT  performance,  parameter 
variability,  and  manufacturing  defect  rates)  for  new 
technologies. 

3.1. 4.7  OUTPUT  -  CRITICAL  FUNCTIONS 
AND  PARAMETERS 

The  supplier  should  create  and  maintain  a  listing  of 
all  critical  system  functions  and/or  parameters. 

These  critical  items  are  identified  from  functional 
analyses  and  Failure  Modes  Effects  or  similar 
analyses.  Customer  inputs  regarding  the  definition 
of  criticality  are  required.  Fault  detection  associated 
with  other  critical  functions  or  parameters  should, 
itself,  be  included  as  a  critical  function. 
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3.1.5  ACTIVITY  -  TRADE-OFF  ANALYSES  BALANCED  DESIGNS 

Trade-Off  analysis  is  a  formal  method  for  making 
cost-effective  decisions  during  Concept  Exploration. 

The  program  must  have  an  identified  list  of  trade 
studies,  the  measures  of  merit  to  be  compared  for 
alternative  technologies  or  system  concepts,  and  a 
weight  factor  for  each  measure  of  merit  that  is 
traceable  to  customer  priorities.  Trade  study 
candidates  are  developed  during  the  Interpretation 
of  Customer  Need  (Activity  3.1.2).  One  or  more  of 
the  reliability  attributes  and  measures  of  merit 
representing  manufacturing  technologies  must  be 
considered  for  all  trade  studies.  The  supplier  must 
have  a  well  defined  Trade-Off  process  that  is  linked 
to  the  Requirements  and  Design  Review  Activity 
(3.1.7). 

3.1 .5.1  INPUT  -  TECHNICAL  OBJECTIVES 

Output  3.1 .2.3  or  3.1 .3.7  serves  as  an  input  to  the 
Trade-Off  Analysis  Activity.  These  are  the 
quantified  baseline  technical  attributes  or  ' 
parameters  that  satisfy  customer  needs  and  include 
a  further  quantification  that  represents  customer 
priorities. 


3.1. 5.2  INPUT  -  TRADE  STUDY 
CANDIDATES 

Output  3.1 .2.5  serves  as  an  Input  to  the  Trade-Off 
Analysis  Activity.  The  list  of  candidates  may 
change  during  the  course  of  the  program  and  should 
be  responsive  to  changes  in  customer  needs  or 
priorities. 

3.1 .5.3  INPUT  -  TECHNOLOGY/CONCEPTS 
ALTERNATIVES  DATA 
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Accurate  information  regarding  the  relationships 
among  technical  attributes  and  parameters  (e.g., 
weight  versus  reliability  attributes)  and  accurate 
descriptions  of  alternative  technologies  or  system 
concepts  is  a  key  to  valid  Trade-Off  conclusions. 

The  claimed  relationships  among  technical  attributes 
and  parameters  should  be  validated  by  experiment 
or  extensive  analysis.  The  data  defining  alternative 
technologies  or  systems  concepts  should 
approximate  the  depth  and  quality  described  in 
paragraph  3.1 .4.2,  Technology/Concept  Data. 
Weighting  factors  assigned  to  the  measures  of  merit 
used  to  compare  alternatives  should  be  traceable  to 
customer  priorities  defined  during  Activity  3.1.2. 

Specific  reliability  and  manufacturing  measures  of 
merit  are  to  be  used  for  all  trade  studies. 

3.1 .5.4  OUTPUT  -  TRADE  STUDY  REPORTS 

These  reports  provide  a  complete  record  of  trade-off 
analysis  results.  This  includes  the  selected 
technology  or  system  concept,  quantitative  values 
of  technical  objectives,  the  trade-study  alternatives, 
methods,  and  selection  criteria.  The  report  should 
describe  how  the  selection  clearly  satisfies 
customer  needs  and  priorities. 


3.1.6  ACTIVITY  -  SYSTEM  REQUIREMENTS 
AND  CONFIGURATION 
RECOMMENDATION 

The  supplier  should  prepare  a  summary  report  of 
Concept  Exploration  phase  activities  that  describes 
the  preferred  system  concept  and  its  functions, 
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quantified  objectives  for  the  system's  reliability 
attributes,  emerging  technologies  proposed  for 
incorporation  in  the  system,  and  risk  reduction  plans 
for  the  riskiest  of  these  technologies. 

3.1 .6.1  INPUT  -  TECHNICAL  OBJECTIVES 

Output  3.1 .2.3  or  3. 1.3.7  serves  as  an  input  to  this 
Activity.  These  data  define  initial  baseline 
quantified  technical  objectives. 

3.1 .6.2  INPUT  -  TECHNOLOGY 
ASSESSMENT  REPORT 

Output  3.1. 4.5  serves  as  an  input  to  this  activity. 

This  input  defines  the  performance  expectations  for 
reliability  attributes,  justified  by  engineering 
rationale  and  supported  by  design  rules. 

3.1. 6.3  INPUT  -  RISK  ASSESSMENT 
REPORT 

Output  3.1 .4.6  serves  as  an  Input  to  Activity  3.1 .6. 
The  data  summarizes  the  risks  associated  with 
achieving  expected  performance  levels  for  the 
reliability  attributes  and  the  plans  for  the  control  of 
these  risks. 

3.1 .6.4  INPUT  -  TRADE  STUDY  REPORTS 

Output  3.1 .5.4  serves  as  an  Input  to  Activity  3.1 .6. 

3.1 .6.5  OUTPUT  -  UPDATED  TECHNICAL 
OBJECTIVES 

Based  on  the  noted  inputs,  the  supplier  must 
synthesize  a  revised  set  of  quantitative  objectives 
for  the  preferred  system  concept  reliability 
objectives.  This  output  should  specify  how  these 
objectives  satisfy  customer  needs  and  identify 
relevant  analyses,  trade  study  reports,  and 
customer  reviews  which  justify  the  decisions. 

3.1 .6.6  OUTPUT  •  RISK  REDUCTION  PLANS 

These  plans  describe  all  moderate  to  high  risk 
technical  issues  and  the  recommended  analyses 
and  tests  required  to  reduce  the  risks  to  low  levels. 
These  plans  define  the  measures  of  merit  that  will  be 
tracked,  the  current  levels  of  these  measures,  and 
the  threshold  values  that  constitute  low  risk.  These 
plans  include  the  definition  of  fall  back  technologies 
and  quantify  the  impact  of  implementing  the  fall  back 
position.  All  technologies  requiring  risk  reduction 
plans  should  include  one  or  more  measures  of  merit 
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dealing  with  reliability  attributes  in  those  plans.  Risk 
Reduction  Plans  specifically  addressing 
achievement  of  customer  objectives  for  reliability 
attributes  may  be  required  based  on  customer 
control  and  audit  results.  (See  paragraph  3.2.) 


3.1.7  ACTIVITY  -  REQUIREMENTS  AND 
DESIGN  REVIEWS 

This  activity  must  maintain  an  essential  discipline 
throughout  the  activities  of  this  phase  and  ensure 
that  the  "voice  of  the  customer"  is  defined  and 
heeded.  The  supplier  should  have  a  documented 
design  review  procedure  that  ensures  consistency 
and  thoroughness.  The  timing  of  Requirements  and 
Design  Reviews  takes  place  during  the  Planning 
Activity  (3.1.3).  The  inputs  to  the  reviewed  activities 
should  serve  to  define  entry  criteria  for  reviews  with 
the  outputs  providing  the  source  of  exit  criteria. 

3.1. 7.1  INPUT  -  ACTIVITY  OUTPUTS 

The  outputs  of  activities  3.1.2  through  3.1.6  should 
be  subject  to  the  review  process.  This  includes  both 
internal  reviews  and  customer  reviews. 

3.1 .7.2  INPUT  -  DESIGN  REVIEW 
PROCEDURES 

The  supplier  must  have  an  established  and 
documented  procedure  for  conducting  internal 
review  of  requirements  and  design.  The  procedure 
covers  review  frequency,  entry  and  exit  criteria, 
definition  of  participants,  agenda  requirements,  and 
closure  plans  for  action  items.  Reviews  should 
always  have  an  agenda  which  defines  the  customer 
needs  to  be  reviewed  and  specific  review  entry  and 
exit  criteria.  With  customer  concurrence,  these 
review  procedures  will  serve  as  the  basis  for 
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customer  reviews.  Every  review  must  include  a 
definition  of  applicable  customer  needs  and  a 
demonstration  of  satisfaction  of  those  needs. 

3.1. 7.3  OUTPUT  •  DESIGN  REVIEW 
REPORTS 

All  design  reviews  must  have  results  documented, 
including  a  definition  of  actions,  responsibility  for 
action  closure,  criteria  for  action  closure,  closure 
dates,  and  final  disposition  of  action  item. 

3.2  CONTROL  AND  AUDIT  QUANTIFY  THE  RISK 

The  control  metrics  for  the  Concept  Exploration 
phase  process  combine  the  Risk  Factor  assessment, 
described  in  paragraph  2.2,  and  an  evaluation  of  the 
quantified  reliability  objectives  contained  in  Output 
3. 1.6.5. 

The  principal  benefits  of  the  Risk  Factor  assessment 
are  as  follows: 

•  Encourages  customer/supplier  involvement. 

•  Requires  an  evaluation  of  the  quality  of  all 
significant  process  activities,  inputs,  and 
outputs  during  this  critical  early  phase  of  the 
acquisition  process. 

•  Provides  credibility  to  quantitative  estimates  of 
reliability  attributes. 

•  Defines  a  numerical  value  representative  of 
the  degree  to  which  the  entire  process  is  in 
control.  These  values  are  used  as  thresholds 
for  corrective  action. 

The  Risk  Factor  assessment  uses  the  process 
described  in  paragraph  2.2  with  the  following 
changes: 

1 )  The  Quality  Evaluation  Activity  (paragraph 
3.1.1)  has  been  added  as  an  activity  requiring 
a  score  representing  its  probability  of  failure. 

2)  The  weight  factor  for  the  Technology 
Assessment  Activity  has  been  changed. 
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3)  The  consequence  of  failure  has  been  modified 

to  account  tor  mission  and  safety  impacts  and 
cost  consequence. 


The  Risk  Factor  assessment  is  conducted  in 
accordance  with  Figures  3-9  through  3-15  (pages 
3-23  through  3-29).  Figure  3-1 0  (and  4-1 0),  which 
provide  the  probability  of  failure  criteria  tor  the 
Quality  Evaluation  Activity,  contain  a  reference  to  a 
‘CAP*  assessment  This  is  the  Certification  and 
Audit  Procedure  contained  in  Volume  II  of  this 
handbook.  The  resultant  Risk  Factor  is  a  score  that 
represents  a  combination  of  the  likelihood  of  failure 
of  the  reliability  process  and  the  consequences  of 
failure. 

If  the  assessment  indicates  that  the  risk  factor  RISK  REDUCTION 

associated  with  achieving  reliability  objectives  is 

too  high,  corrective  action  should  be  taken.  The 

recommended  actions  and  the  associated  Risk 

Factor  scores  are  as  shown  in  Table  3-1 . 


TABLE  3-1 

Risk  Reduction 

Risk  Factor 

Score 

Recommendation 

£  0.4 

None 

>  0.4  -  S  0.7 

Optional  Risk  Reduction  Plan  for 
Reliability 

>  0.7 

Mandatory  Risk  Reduction  Plan 
for  Reliability 

The  customer  is  responsible  for  the  Risk  Factor 
assessment.  Ideally,  the  assessment  is  done  on  a 
continuing  basis  during  the  Concept  Exploration 
phase  based  on  customer/supplier  interaction.  As  a 
minimum  the  information  required  for  the  assessment 
can  be  obtained  during  Design  Reviews. 
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Pretetelllfy  of  Failws 


Activity 


Criteria 


Inadequate  Qualty 


0.1  3.1.1  Figure 

3-10 


Inadequate  Definition  0.3  3.1.2  Figure 

and  Update  of  3.1.7  3-11 

Customer  Requirements 


Fauly  Planning 


Technology  Assessment 


0-2  3.1.3  Figure 

3-12 


0.2  3.1.4  Figure 

3-13 


Inadequate  Trade  Studied  0.2  3.1.5  Figure 

1  3-14 


Cewsqaonts  ef  Fatten 


Weight 


Paraaieter 

c3V2Sr 

c(t) 

Safety  or 
Mission 
Success 
Impact 

Cost  Impact 

C(F) 

0-5C(1|  + 
a5C«> 

rW  I  1  1  I 

Figure  3-9.  Risk  Factor:  Concept  Exploration  Phase 
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Probability 
of  Faffura 


Critiera  for  Activity  3.1.1 


Supplier  has  a  consistent,  recognized  and  demonstrated  history  of  high  quality  and  has  completed  a 
CAP  Assessment  or  Malcolm  Baidrige  application  with  a  s core  above  500. 


Suppfier  has  a  recognized  commitment  to  high  quality,  but  quality  results  are  occasionally  deficient. 
Has  completed  a  CAP  Assessment  or  Malcolm  Baidrige  application  with  a  score  above  350. 


Supplier  commitment  to  quality  is  not  completely  evident  Has  completed  a  CAP  Assessment  or 
Malcolm  Baidrige  application  with  a  score  above  250. 


Supplier  has  completed  a  CAP  Assessment  or  Malcolm  Baidrige  application  with  a  score  above  1 50, 
but  has  no  specific  plans  for  improvement. 


Contractor  has  completed  a  CAP  Assessment  or  Malcolm  Baidrige  application  with  a  score  below  150. 


CC34-007V-2-D 

Figure  3-10.  Probability  of  Failura  P  :  Quality  Evaluation 
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Probability 
of  Faiiurs 

Crftiera  for  Activities  3.1.2  and  3.1.7 

Customer  needs  have  been  identified  and  prioritized  in  a  systematic  manner  and  translated  into 
product  technical  objectives.  Trade  study  candidates  have  been  identified.  Design  reviews  have  clearly 
defined  exit  criteria,  customer  needs  are  reviewed  and  updated,  compliance  with  customer  needs  are 
shown.  Related  research  and  additional  enabling  research  identified  and  documented. 

0.3 

Customer  needs  have  been  identified  in  a  systematic  manner  and  translated  into  product  technical 
objectives.  Trade  study  candidates  have  been  identified.  Customer  needs  reviewed  and  updated  at 
design  reviews.  Compliance  with  customer  needs  are  shown.  Related  research  and  additional  enabling 
research  identified  and  documented. 

0.5 

Customer  needs  have  been  identified  and  translated  into  product  technical  objectives.  Trade  study 
candidate  list  is  incomplete.  Design  reviews  show  compliance  with  customer  needs.  Incomplete 
identification  and  documentation  of  related  research  and  additional  enabling  research. 

0.7 

Customer  needs  only  partially  identified  with  no  clear  traceability  to  product  technical  objectives.  Trade 
study  candidate  list  is  incomplete.  Design  reviews  not  thoroughly  focused  on  customer  needs.  Very 
limited  identification  and  documentation  of  related  research  and  additional  enabling  research. 

0.9 

Limited  customer/supplier  interchange.  Customer  needs  not  validated  with  customer,  design  reviews 
do  not  specifically  address  customer  needs.  No  dear  integration  of  current  research  or  definition  of 
additional  enabling  research. 

CC23-0132-29-O^am 

Figure  3-11.  Probability  of  Failure  :  Customer  Needs  Interpretation 
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Probability 
of  Failure 

Critiara  for  Activity  3.1.3 

0.1 

Funding  profiles  are  Influenced  by  customer  needs  and  priorities.  AH  contract/technical 
requirements  have  been  addressed.  Phase  tasks  are  tailored  to  customer  priorities.  Customer 
needs  dearly  transmitted  to  al  personnel.  Training  needs/plans,  benchmarking  plan  for  continuous 
improvement  identified,  CAE  tools  available  and  data  bases  integrated  Allocated  technical 
objectives  "flowed  down”  and  clearly  traceable  to  customer  needs. 

0.3 

Funding  profiles  are  partially  influenced  by  customer  needs  and  priorities.  ContracVtechnical 
requirements  have  been  addressed.  Customer  needs  not  adequately  transmitted  to  afi  personnel. 
Training  not  specificafiy  addressed.  No  benchmarking  plan.  CAE  tools  list  not  complete.  Allocated 
technical  requirements  partially  traceable  to  customer  needs. 

0.5 

Funding  profiles  not  matched  to  customer  needs  and  priorities.  Contract/technlcaJ  requirements 
have  been  addressed.  Limited  transmittal  of  customer  needs  to  all  personnel  Limited  CAE  tools 
defined  and  data  bases  are  not  Integrated.  A  located  technical  requirements  not  traceable  to 
customer  needs. 

0.7 

Funding  Inadequate.  Contract/technical  re  .  demerits  partially  addressed.  Very  limited  transmittal  of 
customer  needs  to  all  personnel,  very  limited  CAE  tools. 

0.9 

Funding  inadequate.  Contract/technical  requirements  partially  addressed.  No  evidence  of 
transmittal  of  customer  needs  to  personnel.  Extremely  limited  use  of  CAE  tools. 

CC23-0132-30-(>)am 

Figure  3-12.  Probability  of  Failure  P(  :  Program  Planning 
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Crttaria  for  Activity  3.1.4 

To  Determine  the  Total  Scot*.  Add  Indhrldual  Scores  and  Divide  by  Six  (6) 


Mow  Technology  Definition 


New  technologlea/deslgn  concepts  are  fully  described.  Positive  and  negative  features  and  gaps  In  the  knowledge  base  are 
defined.  Expected  operating  environment  del ned  and  traceable  to  customer  mission  description. 


New  technologles/deslgn  concepts  are  partially  described.  Positive  and  negative  features  not  adequately  denned.  Gaps  In  the 
knowledge  base  are  partially  addressed.  Expected  operaring  environment  defined  and  traceable  to  customer  mission 
description. 


New  lechnologlesAjesIgn  concepts  are  partially  described.  Some  positive  features,  few  negative  features  defined.  Limited 
definition  of  gaps  In  the  knowledge  base.  Expected  operating  environment  does  not  consider  maintenance  operations. 


Superficial  description  of  new  technologies/design  concepts.  Negative  features  not  addressed.  No  description  of  gaps  In  the 
knowledge  base.  Limited  description  of  expected  operating  environment. 


New  technologlesAfesIgn  concepts  proposed  without  a  dear  assessment  of  positive  or  negative  features.  No  description  of  gaps 
In  the  knowledge  base.  Limited  description  of  expected  operating  environment 


Expected  Reliability  Performance 


Quantitative  performance  level  estimates  based  on  comparison  to  current  customer  experienced  levels  of  performance  of 
existing  comparable  equipment.  Clear  engineering  rationale  Is  provided  tor  all  expected  Improvements.  Initial  definition  of 
desIgrVappllcaflon  criteria  complete.  Lessons  learned  datals  available  (e.g.,  derating,  environmental  sensitivity). 


Quantitative  performance  level  estimates  based  on  comparison  to  current  levels  ol  performance  ot  existing  comparable 
equipment.  One  or  more  key  performance  parameters  {e.g.  He.  fatigue,  false  alarm  rate,  etc.)  Is  not  Identified.  Clear 
engineering  rationale  is  provided  tor  all  expected  Improvements.  Initial  definition  of  desigiVappilcatlon  criteria  complete.  Lessons 
learned  data  Is  available  (e.g.,  derating,  environmental  sensitivity). 


Quantitative  performance  level  estimates  based  on  comparison  to  current  levels  of  performance  ot  existing  comparable 
equipment.  Key  performance  parameters  missing  and  engineering  rationale  lor  expected  Improvements  Is  weak.  Initial  definition 
ol  oesIgrVappUcatton  criteria  complete.  Lessons  learned  data  is  available  (e.g.,  derating,  environmental  sensitivity). 


Quantitative  performance  level  estimates  have  limited  reference  to  the  performance  levels  of  sxlstlng  comparable  equipment. 
Engineering  rationale  for  expected  Improvements  Is  weak.  Limited  definition  ol  desIgrVappHcatfon  criteria  or  lessons  learned. 


No  definition  ol  existing  comparable  equipment  design/application  criteria  or  lessons  learned. 


Critical  Parametere  or  Fun  effort# 


Failure  modes  effects  analysis  or  similar  analyses  have  been  conducted  to  Identity  critical  parameters  or  functions.  Fault 
detection  and  fault  tolerance  approaches  systematically  defined  and  related  to  critical  parameters  or  functions.  Variability  ot 
critical  parameters  have  been  estimated. 


Failure  modes  effects  analysis  or  similar  analyses  have  been  corxkicted  to  Identify  critical  parameters  or  functions.  Fault 
detection  and  fault  tolerance  approaches  systematically  defined  and  related  to  critical  parameters  or  functions.  Variability  ot 
critical  parameters  not  estimated. 


Failure  modes  effects  analysis  or  similar  analyses  have  been  conducted  to  Identify  critical  parameters  or  functions.  Fault 
detection  and  fault  tolerance  approaches  partially  related  to  critical  parameters  or  (unctions  and  not  accomplished  concurrently 
with  failure  modes  effects  analysis. 


Failure  modes  effects  a-iriysis  or  similar  analyses  have  been  conducted  to  Identify  critical  parameters  or  functions.  No  evidence 
of  linkage  between  this  analysis  and  the  definition  of  fault  detection  and  fault  tolerance  approaches. 


Failure  modes  effects  analysis  or  similar  analyses  not  conducted  or  not  linked  to  identifying  critical  parameters  or  functions,  fault 
detection  or  fault  tolerance  approaches. 


Figure  3-13.  Probability  of  Failure  P^j  Technology  Assessment 
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Criteria  lor  Activity  3.1.4 


ttmnutmcturlng  Technology 


Now  packaging  concepts  and  manufacturing  tochnologlos  Identified.  Sources  ot  variability  have  been  Identified. 


New  packaging  concepts  and  manufacturing  technologies  Identfled.  Variability  not  addressed. 


New  packaging  concepts  and  manufacturing  technologies  only  partially  addressed. 


New  packaging  concepts  parttaty  addressed.  Limited  description  of  manufacturing  technologies. 


Slight  attention  to  new  packaging  concepts  or  manufacturing  technologies. 


fVtk  Reduction  Plant 


All  risk  areas  dearly  identified  (unproven  technologies,  packaging  concepts,  or  manufacturing  technologies,  major  differences 
between  customer  needs  and  the  justified  performance  of  the  equipment).  Risk  reduction  plans  Include  'measures  of  merit"  to 
be  tracked  and  tests  and  analyses  to  be  conducted. 


AH  risk  areas  dearly  Identified  (unproven  technologies,  packaging  concepts,  or  manufacturing  technologies,  major  differences 
between  customer  needs  and  the  justified  performance  of  the  equipment).  Risk  reduction  plans  partially  defined,  missing  one  or 
more  of  the  following:  "measures  of  merit"  to  be  tracked  and  tests  or  analyses  to  be  conducted. 


Risk  areas  not  dearly  Identified  (unproven  technologies,  packaging  concepts,  or  manufactirtng  technologies,  major  dHterences 
between  customer  needs  and  the  justified  performance  of  the  equipment).  Risk  reduction  plans  partially  defined,  missing  one  or 
more  of  the  following:  "measures  of  merit*  to  be  tracked.  Tests  or  analyses  to  be  conducted. 


Few  risk  areas  Identified  (unproven  technologies,  packaging  concepts,  or  manufacturing  technologies,  major  dHterences 
between  customer  needs  and  the  JustHied  performance  of  the  equipment).  Risk  reduction  plans  Incomplete. 


No  risk  reduction  plan. 


Follow  On  nesearcft 


Areas  for  further  research  defined.  Process  In  place  for  Incorporating  these  In  company-wMe  research  programs. 


Areas  for  further  research  defined.  No  specific  process  for  Incorporating  these  In  company-wide  research  programs. 


Areas  for  further  research  partially  defined.  No  specific  process  for  Incorporating  these  In  company-wide  research  programs. 


Areas  for  further  research  partially  Identified.  No  evidence  of  incorporation  In  company-wide  research  programs. 


Limited  definition  of  requirements  for  further  research. 
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Figure  3*13  (Continued).  Probability  of  Failure  P^:  Technology  Aeseesment 
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Probability 
of  Failure 

Criteria  for  Activity  3.1.5 

0.1 

Trade  study  effort  comprehensive.  Candidates  selected  based  on  conflicts  in  satisfying  customer 
needs.  Alternative  technologies/designs  evaluated  in  a  manner  similar  to  that  done  for  the  baseline. 
Trade  study  parameter  weights  are  traceable  to  priorities  identified  during  customer  needs 
interpretation  activity. 

0.3 

Trade  study  effort  comprehensive.  Candidates  selection  not  completely  traceable  to  conflicts  in 
satisfying  customer  needs.  Alternative  technologies/designs  evaluated  in  a  manner  similar  to  that 
done  for  the  baseline.  Trade  study  parameter  weights  are  not  completely  traceable  to  priorities 
identified  during  customer  needs  interpretation  activity. 

0.5 

Trade  study  effort  limited.  Candidate  selection  not  completely  justified.  Alternative 
technologies/desfgns  evaluation  not  as  thorough  as  done  for  the  baseline.  Trade  study  parameter 
weights  are  not  traceable  to  priorities  identified  during  customer  needs  interpretation  activity. 

0.7 

Trade  study  effort  limited.  Candidate  selection  not  justified.  Limited  effort  in  evaluating  alternative 
technologies/designs.  Trade  study  parameter  weights  not  related  to  customer  priorities. 

0.9 

Trade  study  effort  very  limited.  Results  not  credible. 

CC23-0132-33-W|»m 

Figure  3-14.  Probability  of  Failure  P^:  Tradeoff  Analyses 
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1 

1 

Coimqu»nc»» 
of  Failure 
Score 

CHtiora  for  IliMion  Con— quonc—  (C(1^ 

0.1 

Equipment  is  not  significant  to  either  mission  completion  or  sortie  generation  capability. 

0.3 

Equipment  provides  moderate  enhancement  to  mission  success. 

0.5 

Equipment  is  significant  to  mission  success  or  successive  sorties  will  not  be  launched  if 
equipment  is  inoperative. 

0.7 

Item  is  mission  critical.  Failure  will  always  cause  a  mission  abort. 

0.9 

Item  is  safety  critical. 

Consequences 
of  Failure 
Score 

Crttiera  for  Coat  Consequence*  (C{2)) 

0.1 

Equipment  is  simple  (51,000  parts). 

0.3 

Equipment  is  of  minor  complexity  (1,000  -  2,000  parts). 

0.5 

Equipment  is  moderately  compietx  (2,000  -  4,000  parts). 

0.7 

Equipment  is  significantly  complex  (4,000  -  6,000  parts). 

0.9 

Equipment  is  very  complex  (>6,000  parts). 

'  '  CC34 -007V-3-D 

Figure  3-1 5.  Consequence  of  Failure  Criteria 
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The  quantified  reliability  objectives  of  Output  3.1 .6.5 
are  the  other  control  metric(s)  for  the  Concept 
Exploration  Phase.  As  noted  in  paragraph  3.1 .4.5, 
these  projected  values  for  the  reliability  attributes 
(life,  durability,  defect  rate  and  BIT  performance) 
must  be  based  on  a  causal  analysis  of  customer 
experience  data  on  similar  equipment  with 
accompanying  engineering  rationale  for  the 
expected  performance  of  the  proposed  system.  The 
general  process  is  illustrated  by  Figure  3-1 6. 


CC2341 32-26-D 


Figure  3-16.  Reliability  Estimates  Based  on 
Similar  Equipment 


The  quantification  of  performance  based  on 
comparisons  to  similar  equipment  is  the  preferred 
approach  during  the  concept  exploration  phase.  It 
has  the  following  benefits: 

•  Emphasis  on  credible  solutions  to  all  reliability 
problems 

•  Focus  on  customer  use  and  environments 

•  Encourages  customer  involvement 
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•  Consistent  with  the  level  of  detail  available  early 
in  development 

•  Focuses  attention  on  areas  lacking  a  credible 
explanation  regarding  the  satisfaction  of 
customer  needs 

If,  by  the  end  of  the  Concept  Exploration 
phase,  less  than  sixty  percent  (60%)  of  the 
difference  between  current  performance  and 
customer  needs  is  supported  by  credible 
engineering  rationale,  corrective  action 
should  be  required  as  part  of  risk  reduction 
plans. 
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Figure  4-1.  Demonstration  and  Validation  Phase  Activities 
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4.0  INTRODUCTION  PROCESS  OVERVIEW 

This  chapter  defines  the  activities  that  should  take 
place  during  the  Demonstration  and  Validation 
Phase  to  ensure  the  attainment  of  product  reliability 
attributes.  Figure  4-1  shows  the  essential  reliability 
activities  for  the  Demonstration  and  Validation 
Phase  and  the  general  time-phasing  of  these 
activities.  Activity  4.1.1,  Quality  Evaluation,  is 
intended  to  be  a  pre-contract  award  activity. 

Secondly,  for  maximum  benefit  and  efficiency,  much 
of  Activity  4.1.2,  Interpret  Customer  Needs,  should 
take  place  prior  to  either  the  release  of  Requests  for 
Proposal  or  contract  award.  Completion  of  both 
Activity  4.1.2  and  the  remaining  activities  of  the 
phase  takes  place  within  the  framework  of 
contracted  work.  As  described  for  the  prior 
acquisition  phases,  the  Activities  of  this  phase  that 
follow  Program  Planning  are  iterative,  interact  with 
each  other,  and  the  activity  outputs  should  be 
continuously  compared  to  customer  needs.  Detail 
descriptions  of  the  Activities  and  their  Inputs  and 
Outputs  are  provided  in  Paragraphs  4.1.1  through 
4.1.7.  There  are  several  instances  of  commonality 
between  the  tasks  of  this  phase  and  previous 
phases.  In  these  cases,  reference  is  made  to  an 
appropriate  earlier  paragraph  with  any  unique 
description  provided  in  the  paragraphs  of  this 
chapter. 

4.1  SUMMARY  OF  ACTIVITIES  ACTIVITY  SUMMARY 

There  are  two  principal  objectives  for  the 
Demonstration  and  Validation  Phase:  1)  risk 
reduction,  and  2)  EMD  specification  development. 

Technology  issues  that  have  been  identified  as 
offering  substantial  benefits  and  which  have  also 
been  rated  as  moderate  to  high  risk  are  subject  to 
analysis  and  test  to  reduce  their  risk  to  low  levels  for 
entry  into  Engineering  and  Manufacturing 
Development  (EMD).  Fallback  positions  are 
identified  and  developed  for  each  technology  risk 
issue  for  implementation  in  the  event  that  the  risky 
technology  cannot  be  satisfactorily  developed. 

The  performance  requirements  for  the  preferred 
baseline  system  concept  are  refined  through 
analyses,  trade  studies  and  risk  reduction  actions. 
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Firm  performance  and  verification  requirements  are 
established  lor  inclusion  in  EMD  technical 
specifications. 

These  two  themes  are  supported  by  the  reliability 
process  described  in  paragraphs  4.1 .1  through 
4.1 .7.  Validation  of  reliability  attributes  (life, 
durability,  defect  rates  and  BIT  performance)  should 
be  part  of  each  technology  risk  reduction  effort. 
Achievement  of  high  reliability  may,  in  itself,  be 
considered  a  risk  issue  and  subject  to  a  specific  risk 
reduction  plan  during  this  phase.  Additionally, 
quantitative  reliability  performance  and  verification 
requirements,  supported  by  engineering  rationale 
and  actions,  are  developed  for  inclusion  in  EMD 
performance  specifications. 

The  seven  critical  activities  of  this  phase  are: 

1 )  Quality  Evaluation 

2)  Interpret  Customer  Needs 

3)  Program  Planning 

4)  Reliability  Analyses  and  Risk  Reduction 

5)  Trade-Off  Analyses 

6)  EMD  Specification  Development 

7)  Requirements  and  Design  Reviews. 

Most  of  the  activities  are  common  to  those  defined 
for  the  Concept  Exploration  Phase.  The  significant 
differences  are  primarily  based  on  the  level  of 
system  detail  involved  during  Demonstration  and 
Validation  as  compared  to  Concept  Exploration. 

The  Quality  Evaluation  is  used  to  establish  a 
quantified  baseline  for  continuous  improvement. 

The  evaluation  is  conducted  as  part  of  supplier 
proposal  requirements  and  is  pail  of  source 
selection  criteria.  If  suppliers  have  previously 
completed  the  evaluation  as  part  of  the  Concept 
Exploration  Phase  some  improvement  in  the  results 
should  be  expected. 

Accurate  interpretation  of  customer  needs  is  one  of 
the  most  critical  activities  of  this  phase,  as  it  was 
during  the  Concept  Exploration  Phase.  The 
Demonstration  and  Validation  Phase  is  lengthy  and 
some  customer  needs  and  priorities  can  change 
substantially  during  this  phase.  During  the 
Demonstration  and  Validation  Phase,  customer 
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needs,  which  were  translated  to  system  level  design 
requirements  during  Concept  Exploration,  are 
flowed-down  to,  at  least,  the  subsystem  level. 

The  program  planning  activity  is  substantially  the 
same  as  that  done  for  Concept  Exploration.  An 
updated  Risk  Reduction  Plan  is  an  output  that  is 
unique  to  the  Demonstration  and  Validation  Phase. 

Substantial  breadth  and  depth  is  included  in  the 
Reliability  Analysis  and  Risk  Reduction  Activity. 
Environment  and  Use  data  are  more  specifically 
defined  and  translated  into  profiles  representing  life¬ 
time  stresses.  These  are  used  to  both  assess 
quantitative  reliability  attributes  and  define  materials 
characterization  test  requirements.  Technology  risk 
reduction  testing  and  analyses  are  used  to 
substantiate  the  reliability  expectations  of  the 
proposed  new  technologies.  The  definition  of 
critical  functions  and  parameters  are  expanded  to 
lower  levels  of  detail  and  results  are  used  for 
variability  control  analyses.  Proposed 
manufacturing  technologies  and  manufacturing 
processes  are  described.  Critical  processes  are 
identified  and  the  initial  steps  for  process  capability 
determination  are  taken. 

Trade  studies  continue  to  be  the  primary  vehicle  for 
defining  a  balanced  design.  Trade  studies  will  be 
much  more  detailed  and  numerous  than  those 
conducted  during  the  Concept  Exploration  Phase. 
The  studies  will  be  focussed  on  identifying  a  specific 
system  configuration  and  performance  attributes. 

The  ultimate  output  of  the  Demonstration  and 
Validation  Phase  is  a  set  of  technical  specifications 
and  related  documentation  for  EMD.  These  will 
define  both  performance  and  verification 
requirements  for  both  the  system  and,  in  general, 
critical  subsystems.  Related  documentation 
includes  statements  of  work,  supplier  requirements, 
supplier  selection  criteria  and  EMD  risk  reduction 
plans. 

Requirements  and  Design  reviews  bring  discipline 
and  a  focus  on  customer  needs  to  the  Demonstration 
and  Validation  reliability  process.  These  reviews, 
which  include  both  internal  and  external  customer 
reviews  are  a  primary  vehicle  for  continuing  the 
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identification  of  customer  needs  and  the  priorities  of 
those  needs. 


4.1.1  ACTIVITY  •  QUALITY  EVALUATION  EVALUATE  THE 

This  pre-contract  award  Activity  establishes  a  WHOLE 

quantitative  baseline  for  continuous  improvement.  In  ORGANIZATION 

response  to  customer  instructions,  the  supplier 

conducts  an  internal  review  of  company-wide  use  at 

quality  practices.  This  evaluation  uses  criteria 

derived  from  Malcolm  Baldrige  Award  criteria. 

Recommended  criteria  are  provided  in  Volume  II  of 
this  guide. 

This  activity  and  its  Inputs  and  Outputs  are  basically 
the  same  as  defined  in  paragraph  3.1.1  for  Concept 
Exploration.  In  the  conduct  of  this  activity  it  is 
expected  that  each  major  functional  division  (e.g., 

Manufacturing,  Engineering,  etc.)  within  an 
organization  will  assess  their  activities  in  light  of  the 
criteria  contained  in  Volume  II  of  this  guide.  At  the 
organization  level  the  inputs  from  the  functional  level 
are  to  be  aggregated  and  compiled  into  a  single 
report  for  submittal  to  the  customer. 

This  Activity  should  be  applied  to  all  major  contracts 
and  subcontracts.  The  Activity  can  be  integrated 
with  enterprise  wide  actions  such  as  supplier 
certification  programs.  Once  the  activity  has  been 
completed  the  results  should  be  universally 
applicable  to  multiple  contracts. 

4.1. 1.1  INPUT  -  EVALUATION 
INSTRUCTIONS 

The  customer  includes  in  his  request  for  proposal  a 
complete  set  of  instructions  for  conducting  the 
quality  evaluation.  These  include  directions  in 
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Executive  Summaries,  detail  instructions  in  the 
Instructions  to  Offerors  (Evaluation  Criteria, 

Reporting  Requirements),  and  the  relationship  of  toe 
evaluation  to  source  selection  award  factors.  An 
example  of  instructions  is  provided  following 
paragraph  3. 1 . 1 .3.  Report  format  requirements  for  a 
Malcolm  Baldrige  Award  Application  are 
recommended. 

4.1. 1.2  INPUT  -  EVALUATION  CRITERIA 

The  customer  supplies  the  criteria  to  be  used  fo>  the 
quality  evaluation.  Volume  II  of  this  guide  provides  a 
set  of  recommended  criteria  patterned  after  the 
Malcolm  Baldrige  Award  criteria. 

4.1. 1.3  OUTPUT  -  QUALITY  EVALUATION 
REPORT 

The  supplier  provides  a  quality  evaluation  report  in 
accordance  with  customer  reporting  requirements. 
This  report  is  to  be  used  by  both  the  customer  and 
the  supplier.  The  customer  will  score  the  results  and 
use  this  in  source  selection.  The  supplier  should 
similarly  score  his  results  and  use  this  evaluation  to 
define  weaknesses  and  plan  the  required 
improvements. 

If  the  results  of  a  quality  evaluation  have  been 
previously  reported,  for  example  as  a  part  of  the 
Concept  Exploration  Phase,  and  more  than  a  year 
has  elapsed  since  the  evaluation,  a  customer  should 
expect  to  see  improvement  relative  to  the  score  of 
the  previous  report.  Both  the  absolute  score  and 
improvement  should  be  considered  in  the 
application  of  results  to  source  selection. 


4-6 


CHAPTER  4 

DEMONSTRATION  AND  VALIDATION  PHASE 


4.1.2  ACTIVITY  -  INTERPRET  CUSTOMER 
NEEDS 

A  systematic,  comprehensive  process  for  defining 
customer  needs,  the  priorities  associated  with  these 
needs,  and  the  translation  of  these  needs  into 
quantified  technical  objectives  must  be  implemented 
for  this  phase.  Quality  Function  Deployment  is  the 
recommended  tool  for  implementing  this  Activity. 

This  is  one  of  the  most  critical  activities  of  this  phase. 
Customer  contact  and  inputs  should  have  taken 
place  prior  to  contract  award  as  part  of  pre-RFP 
release  discussions.  At  a  minimum,  this  Activity 
must  take  place  as  one  of  the  first  activities  of  the 
Demonstration  and  Validation  Phase.  Customer 
needs  documented  in  contract  technical  material 
should  always  be  supplemented  and  validated  via 
direct  contact.  Customers  must  encourage  and 
support  these  contacts. 

The  translation  of  customer  needs  into  system  level 
technical  requirements  should  have  taken  place 
during  the  Concept  Exploration  Phase.  If  not,  it  must 
be  accomplished  during  this  phase.  Additionally, 
the  flowdown  of  requirements,  via  a  QFD,  to  at  least 
the  subsystem  level  must  take  place  during  the 
Demonstration  and  Validation  Phase.  The  flowdown 
should  continue  to  the  level  of  indenture  required  for 
the  conduct  of  trade  studies  and  other  Demonstration 
and  Validation  Phase  analyses. 

4.1 .2.1  INPUT  -  CUSTOMER  NEEDS  AND 
OBJECTIVES 

Customer  needs  and  objectives  should  be  explicitly 
defined  in  the  contract  technical  documentation, 
such  as  Statements  of  Work  and  preliminary 
specifications.  However,  all  contract  documentation 
must  be  reviewed  for  descriptions  of  customer 
needs.  Additionally,  the  supplier  must  plan  for  early, 
direct,  and  frequent  customer  contact  to  validate  and 
refine  the  documented  needs.  These  contacts  must 
focus  on  a  multidisciplinary  definition  of  customer 
needs. 

4.1. 2.2  INPUT  -  EXPERIENCE  DATA 

The  supplier  must  ensure  that  customer  documented 
needs  are  subject  to  review  by  experts  in  all  areas 
so  that  all  needs  are  identified  and  properly 
translated  into  technical  objectives  and  flowed 
down.  The  supplier  must  also  ensure  that  customer 
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needs  and  technical  objectives,  defined  during  the 
prior  acquisition  phase  is  available  for  application  to 
the  Demonstration  and  Validation  contract.  The 
experience  data  base  should  include  a  thorough 
and  documented  understanding  of  both  good  and 
bad  customer  experience  with  current  products  as  a 
preferred  method  for  aiding  in  the  identification  of 
needs. 


4.1. 2.3  OUTPUT  -  TECHNICAL 
OBJECTIVES 

The  supplier  must  provide  a  definition  of  specific 
technical  attributes  and  parameters  that  will  satisfy 
all  customer  needs.  Quality  Function  Deployment  is 
the  recommended  method  for  defining  and 
documenting  these  technical  objectives.  This 
definition  includes  quantitative  objectives  for  each  of 
the  attributes  and  parameters.  The  customer  needs 
must  be  given  weighting  factors  which  represent 
priority  levels.  These  factors  will  be  used  to  define 
the  relative  importance  of  each  of  the  technical 
attributes  and  parameters.  It  is  critical  that  the 
interaction  between  reliability  technical  attributes 
and  the  other  technical  attributes  be  identified  for  the 
purpose  of  identifying  trade  study  candidates. 

The  technical  objectives  should  be  flowed  down  by 
QFD  procedures  to  the  subsystem  and  lower  levels 
of  indenture.  The  flowdown  must  maintain 
traceability  to  customer  needs  and,  more 
importantly,  the  customer’s  priorities  for  these  needs. 
In  order  to  retain  focus  on  an  integrated  set  of 
reliability  requirements,  it  is  recommended  that  all 
the  product  reliability  attributes  be  grouped  together 
during  the  QFD  development.  These  attributes 
include  service  life,  durability,  defect  rates,  and  BIT 
performance  requirements  (e.g.,  False  Alarm  rate, 
Fault  Detection  Probability,  Could  Not  Duplicate 
(CND)  rate,  etc.). 

Early  in  the  Demonstration  and  Validation  Phase 
these  quantified  attributes  and  parameters  should  be 
treated  as  objectives  rather  than  requirements.  As  a 
subset  of  this  output,  the  supplier  must  identify,  list, 
and  update  all  related  research  that  supports  the 
system  development.  This  list  includes  any  research 
not  otherwise  a  part  of  risk  reduction  plans  in  order  to 
avoid  duplication  and  maximize  resources.  As  the 
Demonstration  and  Validation  Phase  proceeds, 
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additional  research  applicable  to  the  system  must  be 
identified  for  inclusion  in  future  plans. 

4.1 .2.4  OUTPUT  -  TRADE  STUDY 
CANDIDATES 

The  supplier  must  identify  candidates  for  trade 
studies  to  be  performed  during  the  Demonstration 
and  Validation  Phase.  The  Quality  Function 
Deployment  techniques  can  assist  in  defining  these 
candidates  by  evaluating  the  interactions  among 
technical  attributes.  Trade-off  analyses  are  used  to 
optimize  a  technical  attribute  or  parameter  that 
interacts  with  other  technical  attributes  or 
parameters.  For  example,  radar  detection  range  is  a 
function  of  power,  weight,  volume,  receiver 
sensitivity,  reliability,  and  cost.  Trade  studies  are 
also  used  to  select  between  alternative  technologies 
or  system  concepts. 


4.1.3  ACTIVITY  -  PROGRAM  PLANNING 

The  Program  Planning  Activity  description  is  the 
same  as  for  the  Concept  Exploration  Phase 
(paragraph  3.1.3)  with  appropriate  changes  in  the 
reference  paragraph  numbers.  Demonstration  and 
Validation  Phase  planning  considers  one  additional 
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input,  Risk  Reduction  Plan,  and  generates  one 
additional  output,  an  Updated  Risk  Reduction  Plan. 

The  planning  is  also  more  detailed  than  that  done  tor 
Concept  Exploration  as  a  result  of  the  greater  depth 
of  technical  detail  and  the  longer  program  duration. 

4.1. 3.1  INPUT  -  FUNDING  PROFILE 

The  description  of  this  Input  is  the  same  as  provided 
in  paragraph  3. 1.3.1. 

4.1. 3.2  INPUT  -  CONTRACT 
REQUIREMENTS 

The  description  of  this  Input  is  the  same  as  that 
provided  in  paragraph  3. 1.3.2. 

4.1 .3.3  INPUT  -  RISK  REDUCTION  PLANS 

Output  3.1 .6.6  serves  as  an  Input  to  the  Program 
Planning  Activity.  Risk  Reduction  Plans  are 
mandatory  for  the  Demonstration  and  Validation 
Phase.  They  should  be  prepared  as  part  of  the 
Concept  Exploration  Phase. 

4.1 .3.4  INPUT  -  TECHNICAL  OBJECTIVES 

Output  4.1 .2.3  serves  as  an  input  to  the  Program 
Planning  Activity. 

4.1 .3.5  INPUT  -  TRADE  STUDY 
CANDIDATES 

Output  4. 1.2.4  serves  as  an  input  to  the  Program 
Planning  Activity.  Additional  trade  study  candidates 
may  have  been  developed  as  a  result  of  risk 
reduction  plans,  customer  requests,  or  supplier 
experience.  The  candidate  list  should  define  the 
specific  measures  of  merit  being  used  to  evaluate 
trade  study  alternatives.  Any  weighting  factors 
applied  to  the  trade  study  measures  of  merit  should 
be  shown  in  the  candidate  list  and  should  be 
traceable  to  customer  priorities.  Reliability  attributes 
and/or  parameters  should  be  considered  for  all  trade 
studies  as  measures  of  merit. 

4.1. 3.6  OUTPUT  -  TAILORED  PROGRAM 
PLAN 

The  description  for  this  Output  is  the  same  as 
provided  in  paragraph  3. 1.3.5  with  appropriate 
changes  made  to  the  paragraphs  referenced  therein. 
The  tailored  program  plan  should  include  inputs  from 
relevant  functional  disciplines,  e.g.  Reliability, 
Design  Engineering,  Diagnostics  Design, 
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Manufacturing,  Engineering  Technologies  (e.g. 
Structural  Dynamics,  Thermodynamics,  etc.),  and 
Logistics. 

4.1 .3.7  OUTPUT  •  UPDATED  RISK 
REDUCTION  PLAN 

The  baseline  Risk  Reduction  Plan  should  be 
reviewed  and  modified  based  on  changes  in 
customer  needs,  risk  assessment  results 
(identification  of  technology  risk  levels),  proposed 
technologies,  and  resource  constraints.  Specific 
attention  should  be  paid  to  including  new 
manufacturing  technologies  as  risk  reduction 
candidates.  The  basic  attributes  of  the  Risk 
Reduction  Plans  remain  as  described  in  paragraph 
3.1. 6.6. 

A  principle  subsection  of  the  Updated  Risk 
Reduction  Plan  should  describe  a  summary  test  plan 
with  requirements  for  Failure  Reporting  Analyses 
and  Corrective  Action  System  (FRACAS).  All 
testing  conducted  during  the  Demonstration  and 
Validation  Phase  should  be  considered  risk 
reduction  testing.  The  test  plan  summary  should 
describe  how  test  results  can  be  used  to  validate  the 
reliability  attributes  of  the  risky  technology,  define 
environmental  stress  properties/sensitivities,  and 
identify  variability  associated  with  the  critical 
parameters  of  the  tested  items.  The  FRACAS  plan 
should  describe  test  objectives,  failure  criteria,  the 
responsibility  for  root  causal  analyses  and 
correction,  and  the  process  for  ensuring  that  these 
lessons  learned  are  translated  into  design  rules, 
application  guides,  or  derating  criteria. 

4.1 .3.8  OUTPUT  ■  TECHNICAL 
OBJECTIVES  FLOWDOWN 

The  description  of  this  output  is  the  same  as 
provided  in  paragraph  3. 1.3.7.  The 
flowdown/allocation  of  reliability  technical 
objectives  needs  to  be  done  to  the  level  of  detail  at 
which  the  Demonstration  and  Validation  analyses 
and  trade  studies  will  be  conducted.  Allocations 
should  also  be  established  for  any  subcontract  effort. 

4.1. 3.9  OUTPUT  -  SUPPLIER  SELECTION 
AND  CONTROL  PLAN 

The  description  of  this  output  is  the  same  as 
provided  in  paragraph  3.1 .3.9  with  the  following 
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ackfitions: 

1 )  The  reliability  and  quality  criteria  used  for 
supplier  selection  should  be  identified. 

2)  Criteria  for  defining  screening  testing  that 
would  be  applied  to  delivered  supplier 
equipment 

4.1.3.10  OUTPUT  -  BENCHMARKING  AND 
TRAINING  PLAN 

The  description  of  this  output  is  the  same  as 
provided  in  paragraph  3. 1.3.8.  Some  potential 
candidates  for  Benchmarking  are: 

•  Design  for  Manufacturing 

•  Design  for  Testability 

•  Trade  Study  Processes 

•  Risk  Analyses 

•  Design  for  Reliability 

•  Design  Automation 

Emphasis  on  benchmarking  as  an  effective  process 
improvement  tool  must  begin  early  in  the  acquisition 
cycle. 


4.1.3.11  OUTPUT  -  CAE  TOOLS 

The  description  of  this  output  is  the  same  as 
provided  in  paragraph  3. 1.3.6. 


CONTINUOUS 
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4.1.4  ACTIVITY  -  RELIABILITY  ANALYSIS 
AND  RISK  REDUCTION 

This  is  the  central  Activity  of  the  Demonstration  and 
Validation  reliability  process.  The  Activity  is 
continually  iterated  throughout  the  phase  as  detail 
information  is  developed  through  analysis,  test,  and 
trade-off  analyses.  There  are  two  basic  elements  of 
this  Activity: 

1 )  Development  of  design  and  manufacturing 
criteria  and  application  rules  that  will  prevent 
defects  in  both  the  new  and  existing 
technologies  proposed  for  systems  use. 

2)  Estimation  of  quantitative  reliability  parameters, 
based  on  the  above,  for  inclusion  in  EMD 
technical  specifications. 

The  key  implementation  issue  for  a  supplier  is  the 
integration  and  coordination  of  the  efforts  of  multiple 
functional  specialties  which  affect  defect  prevention 
and  elimination.  The  descriptions  of  inputs  and 
outputs  for  this  Activity  could  be  used  to  establish 
checklists  directed  at  achieving  the  required 
coordination. 
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4.1 .4.1  INPUT  -  SYSTEM  DESIGN  DATA 

These  data  provide  a  complete  engineering 
description  of  the  proposed  system  design.  It  should 
include  the  following: 

•  Functional  analysis  results,  functional  block 
diagrams,  and  schematics  of  both  the  electrical 
and  mechanical  design  (e.g.  packaging,  cooling) 
of  the  system/equipment.  The  functional 
analyses  should  show  time  phased  functions  for 
all  operational  use  including  routine  checkout 
and  maintenance. 

•  Description  of  new  technology  (parts,  materials, 
assemblies)  including  known/suspected 
environmental  sensitivity,  critical  parameters  with 
expected  variability,  status  of  development 
testing,  reference  to  risk  reduction  plans,  and 
packaging  concepts. 

•  Description  of  existing  technology  being  applied 
to  the  system. 

•  Description  of  comparable  current  technology 
and  systems/equipment  that  can  be  used  as  a 
basis  for  quantitative  reliability  estimates 
applicable  to  the  new  system.  The  data  is  to 
include  a  definition  of  operating  and  maintenance 
environments  and  functional  operation. 

4.1 .4.2  INPUT  -  TECHNICAL  OBJECTIVES 

Output  4. 1.3.8  serves  as  an  input  to  this  analysis 
activity.  These  are  the  baseline  set  of  quantitative 
reliability  expectations. 

i4.1.4.3  INPUT  -  EXPERIENCE  DATA 

These  data  include  the  following: 

•  Existing  de-rating,  application,  design  rules, 
lessons  learned  (internal  and  customer),  design 
for  manufacturing  rules,  customer  use  reliability 
data  for  the  systems  or  equipments  identified  in 
Input  4.1 .4.1  and  manufacturing  defect  data  for 
current  manufacturing  processes. 

•  Data  from  related  research  for  the  proposed  new 
technologies  and  recommended  application/de¬ 
rating  guides. 
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This  input  provides  an  opportunity  lor  significant 
supplier/  customer  interaction  during  the 
development  of  user  experience  data  on  current, 
comparable  systems  or  technology. 

4.1. 4.4  INPUT  -  ENVIRONMENT  AND  USE 
DATA 

The  supplier  is  responsible  for  defining  all  local 
environmental  conditions  and  resultant 
stresses/stress  cycles.  The  data  must  be  traceable 
to  the  customer  provided  mission  description  (or 
similar  document).  Relevant  information  includes 
basing  location  and  operating  mission  profiles. 
Internally  generated  stresses  should  be  derived  from 
the  functional  analyses  of  Input  4.1 .4.1 .  Stresses 
generated  as  a  result  of  maintenance,  storage, 
handling,  transportation  and  manufacture  must  also 
be  derived.  The  data  should  show  expected 
maximum  stress  conditions  and  life  cycle  stress 
profiles,  showing  amplitude  and  frequency,  based 
on  the  distribution  of  basing  locations  and 
types/duration  of  missions  and  other  operation/use. 
Analyses  should  focus  on  primary  fault  producing 
stresses  such  as  vibration,  loads,  shock, 
temperature  and  temperature  cycling,  humidity,  and 
handling  stresses. 

4.1 .4.5  INPUT  -  RISK  REDUCTION  RESULTS 

The  supplier  is  responsible  for  ensuring  that  the 
results  of  risk  reduction  tests  and  analyses  are  used 
to  develop  the  design  and  manufacturing  guides 
necessary  to  prevent  defects  in  new  or  risky 
technology.  Tests  may  include  materials  and  parts 
properties  characterization  defining  responses  to 
stress  and  stress  cycling,  the  identification  of  parts 
and  materials  critical  parameter  variability  and  likely 
control  factors.  Analysis  results  may  include  failure 
rate,  testability  attributes,  and  fatigue  analyses. 

4.1 .4.6  INPUT  •  MANUFACTURING 
PROCESS  DESCRIPTION 

The  supplier  should  develop  initial  step  by  step 
descriptions  of  the  projected  manufacturing 
processes.  These  should  focus  on  new  processes 
required  for  new  parts/materials/packaging 
technologies  and  any  new  manufacturing 
technology  intended  as  an  improvement  to  existing 
manufacturing  processes.  These  descriptions 
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should  identify  any  known  constraints  placed  on 
system  electrical  or  mechanical  design  or  unique 
stresses  imposed  on  the  system  elements.  The 
inputs  should  address  the  definition  of  capability 
indices  for  both  existing  and  new  manufacturing 
processes  and  identify  any  risk  reduction  analysis  or 
tests. 


4.1. 4.7  OUTPUT  -  DESIGN  CRITERIA 
REPORT 

The  supplier  is  responsible  for  generating  a 
comprehensive  report(s)  describing  the  design  and 
manufacturing  criteria  essential  to  preventing 
product  defects.  This  report  should  form  the  basis  for 
the  development  of  design  and  manufacturing 
guides  and  specifications.  The  report  also  provides 
engineering  justification  for  the  quantitative 
estimates  of  Output  4.1 .4.9.  The  topics  covered  by 
the  report  should  address: 

•  Fatigue  and  durability  design  requirements 

•  Environmental  stress  protection 

•  Electrical,  mechanical,  and  thermal  design  rules 

•  Parts  and  materials  applications  guides 

•  BIT  and  Diagnostics  design  guides 

•  Lessons  Learned 

•  Parts  and  materials  selection  criteria 

•  Design  for  manufacturing  rules 

The  above  criteria  should  pay  specific  attention  to 
new  technologies.  The  material  is  developed  from 
existing  criteria,  risk  reduction  tests  and  analyses, 
and  other  Demonstration  and  Validation  stress  and 
fatigue  analyses. 

4.1 .4.8  OUTPUT  -  CRITICAL  FUNCTIONS, 
PARAMETERS,  AND  PROCESSES 

This  output  product  represents  the  initial  steps  of  a 
worst  case/variability  control  program.  During 
Demonstration  and  Validation  all  critical  functions 
should  be  identified  by  a  combination  of  Failure 
Modes  Effects  Criticality  Analysis  (or  similar 
analyses),  engineering  judgement  and  experience. 
The  criticality  of  these  functions  should  be 
coordinated  with  the  customer  who  is  the  final  arbiter 
of  criticality.  The  criticality  of  functions  should  then 
be  flowed-down  through  the  system  to  identify 
critical  subsystems  and  assemblies.  The  design  of 
the  BIT  and  diagnostic  system  should  be  driven,  or 
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at  least  prioritized,  by  this  criticality  analysis,  as 
should  the  design  of  fault  tolerance  features. 

Additional  critical  parameters  can  be  identified  for 
new  technologies  by  the  results  of  risk  reduction 
tests  and  analyses  and  should  include  critical 
mechanical  functions  such  as  thermal  control.  Once 
critical  parameters  have  been  identified,  the  results 
are  compared  to  the  manufacturing  process 
description  (Input  4.1 .4.6)  to  define  critical 
manufacturing  processes  that  are  potential 
candidates  for  both  variability  reduction  techniques 
and  statistical  process  control.  Critical 
manufacturing  process  steps  such  as  soldering 
operations  should  be  identified  based  on  experience 
and  judgement  regarding  their  overall  impact  on 
product  reliability. 

Once  the  critical  parameters  have  been  identified 
their  variability  should  be  established  through  test  or 
analysis.  CAE  tools  and/or  circuit  simulation  tools 
can  be  used  to  evaluate  these  variances  against 
performance  specification  limits.  This  analysis  has 
significant  benefits  in  the  design  of  properly 
operating  BIT  and  diagnostic  functions. 

4.1 .4.9  OUTPUT  -  QUANTITATIVE 
RELIABILITY  EXPECTATIONS 

Quantitative  reliability  expectations  are  to  be 
estimated  based  on  comparisons  to  the  user 
experienced  performance  of  current  similar  systems 
or  equipment.  The  reliability  parameters  of  interest 
are:  service  life,  defect  rate  (e.g.  Mean  Time 
Between  Maintenance,  to  include  all  defects  such 
as  fatigue,  wearout,  catastrophic  failure,  degraded 
performance,  alleged  overstress  and  faulty  BIT 
performance),  Mission  Success  Probability,  and  BIT 
performance.  Causal  analyses  should  be  done  on 
the  current  performance  data  to  establish  rationale 
for  those  levels  of  performance.  Current 
performance  data  provided  by  Input  4. 1.4.3  should 
include  manufacturing  defect  data  from  which 
conclusions  regarding  it's  impact  on  field 
performance  can  be  drawn.  Inputs  4. 1.4.1  and 
4. 1.4.4  provide  the  data  required  for  evaluating 
differences  in  environments  and  use.  Combining 
these  factors  with  data  representing  new  technology 
characteristics  derived  from  risk  reduction  activities 
and  the  defect  prevention  actions  described  in 
Outputs  4.1. 4.7  and  4.1. 4.8  should  permit  an 


CUSTOMER  BASED 

RELIABILITY 

ESTIMATES 


4-17 


CHAPTER  4 

DEMONSTRATION  AND  VALIDATION  PHASE 


assessment  of  reliability  based  on  solid  engineering 
analyses. 

Initial  life  estimates  for  known  fatigue  or  wearout 
mechanisms,  particularly  lead  or  solder  joint  failure, 
should  be  made  using  maximum  derived  stress 
cycling  applied  at  worst  case  locations. 

Analyses  of  the  proper  performance  of  BIT  systems 
should  be  done  using  circuit  simulation  analyses. 


4.1.5  ACTIVITY  -  TRADE  OFF  ANALYSIS 

The  description  of  this  Activity  is  the  same  as 
provided  in  paragraph  3.1.5.  Key  elements  are:  the 
existence  of  a  well-defined  trade  off  procedure,  the 
traceability  of  weighting  factors  for  measures  of  merit 
to  customer  priorities  identified  in  paragraph  4.1.2, 
and  the  inclusion  of  reliability  and  manufacturing 
parameters  in  all  trade  studies. 

4.1 .5.1  INPUT  -  TRADE  STUDY 
CANDIDATES 

Output  4. 1.2.4  serves  as  an  input  to  the  Activity. 

This  input  is  a  summary  description  of  each  of  the 
trade  studies  to  be  conducted  during  the 
Demonstration  and  Validation  phase.  The  summary 
includes  the  trade  study  issue,  definition  of 
alternatives,  parameters  to  be  considered  in 
reaching  a  conclusion  and  responsibilities  for 
conducting  the  study. 


4-18 


CHAPTER  4 

DEMONSTRATION  AND  VALIDATION  PHASE 


4.1. 5.2  INPUT  -  TECHNICAL  OBJECTIVES 
FLOWDOWN 

Output  4.1 .3.8  defines  the  baseline  quantitative 
reliability  objections  that  satisfy  customer  needs  and 
serves  as  an  Input  to  the  Trade-Off  Activity. 

4.1. 5.3  INPUT  -  DESIGN  DATA 

The  description  of  this  input  is  the  same  as  contained 
in  paragraph  3. 1.5.3  with  the  referenced  paragraphs 
changed  from  3.1 .4.2  and  3.1 .2  to  4.1. 4.4  and  4.1.2. 

4.1. 5.4  OUTPUT  -  TRADE  STUDY  REPORTS 

The  description  of  this  output  is  the  same  as 
contained  in  paragraph  3.1 .5.4.  The  supplier  trade 
study  procedure  should  identify  the  specific 
mechanisms  for  ensuring  that  changes  resulting  from 
trade  studies  are  formally  implemented  and 
information  regarding  these  changes  is  disseminated 
throughout  the  program. 


4.1.6  ACTIVITY  -  EMD  SPECIFICATION  DEFINE  THE 

DEVELOPMENT  REQUIREMENTS 

Creation  of  EMD  technical  specifications  is  one  of 
the  primary  objectives  of  the  Demonstration  and 
Validation  Phase  activities.  These  will  contain 
requirements  for  the  following  issues: 

1)  Quantative  reliability  requirements  (life, 
success  probabilities,  fault  rate,  and  BIT 
performance) 
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2)  Design  criteria  and  parts  selection, 
standardization  requirements 

3)  Variability  requirements 

4)  Test  and  verification  requirements 

All  of  the  above,  with  the  exception  of  Test  and 
Verification  requirements,  should  be  developed  from 
the  Input  and  Output  products  of  the  previous 
Activities. 

This  Activity  also  includes  the  development  of 
Statements  of  Work  and  Data  Item  descriptions. 
Specifications,  Statements  of  Work,  Data  Item 
requirements,  and  supplier  selection  criteria  should 
also  be  developed  for  major  subcontracted  items. 

These  will  continue  a  flowdown  of  the  requirements 
developed  for  the  principal  customer. 

4.1. 6.1  INPUT  -  TECHNICAL  OBJECTIVES 
FLOWDOWN 

Output  4.1 .3.8  serves  as  an  input  to  this  Activity. 
These  represent  the  baseline  quantitative  reliability 
objectives  that  are  traceable  to  the  satisfaction  of 
customer  needs. 

4.1. 6.2  INPUT  -  QUANTITATIVE 
RELIABILITY  EXPECTATIONS 

Output  4.1. 4.9  serves  as  an  input  to  this  Activity. 
These  reliability  estimates  are  the  results  of  Activity 
4.1.4  (Reliability  Analysis)  and  4.1.5  (Trade-Off 
Analysis).  These  results  must  be  compared  to  the 
baseline  objectives  to  define  EMD  specification 
requirements.  The  baseline  values  should  be 
changed,  subject  to  customer  approval,  when  it  is 
demonstrated  that  a  properly  balanced  design 
results  in  a  reduction  of  certain  reliability  objectives. 
The  comparison  of  baseline  values  to  estimates  is 
otherwise  used  to  establish  the  level  of  risk 
associated  with  meeting  customer  needs  and 
provide  a  basis  for  EMD  risk  reduction  plans. 

4.1. 6.3  INPUT  -  ENVIRONMENT  AND  USE 
DATA 

Input  4.1. 4.4  also  serves  as  an  input  to  this  Activity. 
These  data  serve  to  establish  design-to  requirements 
and  the  conditions  under  which  EMD  verification 
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testing  should  take  place. 

4.1. 6.4  INPUT  -  DESIGN  CRITERIA 

Output  4.1 .4.7  serves  as  an  input  to  this  Activity. 
Parts  selection,  derating,  design  margins  for  the 
EMD  specifications  are  determined  from  this  output. 
This  output  (4. 1.4.7)  also  provides  a  source  for 
developing  task  requirements  for  statements  of  work. 

4.1 .6.5  OUTPUT  -  EMD  SPECIFICATIONS 

This  output  represents  the  entire  range  of  technical 
requirements  products:  customer  specifications  and 
statements  of  work,  supplier  specifications  and 
statements  of  work,  and  supplier  selection  criteria. 
The  QFD  used  to  translate  customer  needs  into 
technical  objectives  and  the  subsequent  flowdown 
QFD’s  should  serve  as  the  basis  for  identifying 
specification  performance  issues.  A  comparison  of 
baseline  quantitative  objectives  with  estimates 
derived  from  Demonstration  and  Validation  analyses 
and  Trade-Off  analyses  provides  the  data  required 
to  define  EMD  specification  quantitative  values.  The 
design  criteria  input  should  be  used  to  define 
specification  requirements  for  design  margins  while 
design-to  and  test  environments  are  determined 
using  the  environment  and  use  data  input.  These 
data  should  be  structured  to  describe  a  life  use 
profile  (frequency  and  amplitude  of  stress  over  the 
life  cycle  of  the  product)  for  application  to  EMD 
Reliability  development  and  verification 
requirements. 

EMD  verification  and  testing  includes  analytical 
verification,  development  tests,  verification  tests, 
and  Environmental  Stress  Screening  and 
Acceptance  tests.  Current  literature  contains 
sufficient  information  to  develop  the  appropriate  test 
detail  for  all  testing.  The  essentia!  element  of 
development  and  verification  tests  is  that  they  be 
conducted  under  the  customer  use  environment. 
These  may  be  accelerated  using  recognized 
environmental  acceleration  factors. 

Testing  must  include  the  validation  of  life  and  fatigue 
characteristics.  These  can  be  done  with  mockups  or 
representative  samples. 
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Development  testing  includes  any  required  material 
characterization  test  that  defines  behavior  under 
stress  cycling  or  extremes,  electrical  performance 
including  BIT  performance,  critical  parameter 
variability  definition  and  manufacturing  process 
characterization. 

Statement  of  Work  task  development  is  also  part  of 
this  output.  These  tasks  include  a  description  of 
EMD  analytical  requirements  and  can  be  developed 
using  the  EMD  process  description  contained  in 
Chapter  5  herein. 

Specifications  and  Statements  of  Work  for  major 
EMD  subcontracted  equipment  should  also  be  part 
of  the  output  described  herein.  QFD  evaluation  can 
be  used  to  flowdown  technical  requirements  to  the 
appropriate  level.  Task  requirements  for  Statements 
of  Work  are  replications  of  the  tasks  required  of  the 
prime  contractor,  tailored  to  the  details  and 
importance  of  the  subcontracted  equipment. 

4.1. 6.6  OUTPUT  -  EMD  RISK  REDUCTION 
PLAN 

Reliability  risk  reduction  plans  are  necessary  for  any 
Demonstration  and  Validation  Phase  risks  that  have 
not  been  satisfactorily  reduced  to  a  low  level,  any 
additional  risks  uncovered  during  Demonstration 
and  Validation,  or  as  determined  by  the  Control  and 
Audit  requirements  of  paragraph  4.2.  These  plans 
should  include  the  criteria  for  identifying  risks,  the 
risks  that  have  been  identified  for  EMD  closure,  the 
measures  of  merit  to  be  tracked  to  verify  risk  closure, 
the  analyses  and  tests  required,  and  the 
recommended  fall  back  positions.  Inputs  4.1. 6.2  and 
4.1. 6.4  are  primary  sources  for  identifying  risk 
reduction  candidates.  t 


CONTINUE  RISK 
REDUCTION 
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4.1.7  ACTIVITY  -  REQUIREMENTS  AND 
DESIGN  REVIEWS 

The  description  of  this  Activity  is  the  same  as 
provided  in  paragraph  3.1.7.  Design  reviews  should 
always  be  focused  on  specific  customer  reliability 
need  as  identified  and  translated  into  technical 
objectives  by  Activity  4.1.2  and  the  status  of  the 
design  in  satisfying  those  needs.  The  Risk  Factor 
assessment  figures  contained  in  paragraph  4.2  can 
be  used  to  establish  checklists  for  requirements  and 
design  review  topic  material. 

4.1. 7.1  INPUT  -  ACTIVITY  OUTPUTS 

All  outputs  of  Activities  4.1.2,  4.1.3,  4.1.4,  4.1.5,  and 
4.1.6  are  the  issues  for  both  internal  and  customer 
reviews.  Certain  outputs  such  as  those  from  the 
Program  Planning  and  EMD  Specification  Activities 
(4.1.3  and  4.1 .6)  are  subject  to  a  limited  number  of 
reviews  at  the  start  and  near  the  end  of  the 
Demonstration  and  Validation  Phase.  The  output  of 
Activity  4.1.2  serves  as  the  benchmark  against 
which  the  outputs  of  Activities  4.1 .4  and  4.1.5  are 
measured.  The  outputs  of  Activities  4.1.4  and  4.1 .5 
provide  the  technical  material  that  is  subject  to 
continuous  review  during  the  Demonstration  and 
Validation  Phase. 

4.1. 7.2  INPUT  -  DESIGN  REVIEW 
PROCEDURES 

The  description  of  this  input  is  the  same  as  provided 
in  paragraph  3. 1.7.2. 

4.1. 7.3  OUTPUT  -  DESIGN  REVIEW 
REPORTS 

The  description  of  this  output  is  the  same  as 
provided  in  paragraph  3.1 .7.3.  These  reports  are  the 
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vehicles  for  documenting  changes  in  customer 
requirements. 

4.2  CONTROL  AND  AUDIT  CONTROL  THE 

PROCESS 

During  the  Demonstration  and  Validation  Phase  of 
the  acquisition  cycle,  as  in  all  other  phases,  teaming 
of  multiple  disciplines  is  essential  in  developing  a 
product  that  will  satisfy  reliability  needs  of  the 
customer.  It  is  during  this  phase  that  the  design 
concept  is  developed  into  viable  product 
specifications.  Heavy  emphasis  is  placed  on 
developing  the  design  to  meet  all  reliability 
requirements.  As  the  design  takes  shape,  numerous 
tools  and  techniques  are  used  in  a  process  that  has 
high  probability  of  producing  a  product  that  is  very 
reliable.  Initial  reliability  predictions  are  based  on 
customer  use  data  and  lessons  learned  and  updated 
as  the  product  takes  shape. 

To  ensure  that  processes  that  will  increase  product 
reliability  exist  and  that  these  processes  are  applied, 
a  continuation  of  the  Risk  Factor  assessment 
described  in  Chapters  2  and  3  is  applied. 

During  the  Dem/Val  Phase  the  contractor  must  have 
the  following  controls  in  place: 

•  An  effective  system  that  translates  all  the 
customer’s  expectations  and  requirements  into 
product  design  and  performance  requirements 
and  provides  for  continuous  review  of  the 
satisfaction  of  customer  needs. 

•  A  risk  management  system  for  the  evaluation  and 
elimination  of  technical  and  reliability  risk. 

•  The  necessary  processes  to  ensure  that  the 
design  is  assessed  for  product 
reliability/durability. 

•  A  process  that  addresses  issues  that  will  ensure 
a  robust  design  and  manufacturing  approach. 

•  A  system  that  will  provide  effective  process 
controls  for  identifying  key  product  and  process 
characteristics. 
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•  A  defined  baseline  for  continuous  improvement 
and  processes  for  achieving  it. 

The  evaluation  of  the  reliability  process  is  based  on 
assessing  these  control  indicators  and  the  degree  to 
which  they  are  implemented  into  the  contractor’s 
way  of  doing  business.  Figure  4-9  (page  4-26) 
summarizes  the  assessment  of  the  Demonstration 
and  Validation  Phase;  Figures  4-10  through  4-15 
(pages  4-26  through  4-32)  describe  the  criteria  for 
evaluating  each  process  element  and  the 
consequences  of  failure  of  the  process. 

The  risk  factor  evaluation  is  a  customer 
responsibility.  The  evaluations  should  use  a 
multidiscipline  team  and  can  be  conducted  as  part  of 
design  reviews.  A  Demonstration  and  Validation 
Phase  is  typically  three  to  four  years  in  length.  Asa 
resutt,  the  risk  factor  evaluation  should  be  Iterated 
several  times  throughout  the  phase.  A 
recommended  schedule  for  these  evaluations  is: 
initial  evaluation  at  six  months  after  oontract  award, 
semiannual  thereafter  with  a  final  assessment  six 
months  prior  to  completion  of  tie  conked 

If  the  assessment  indicates  that  the  risk  factor  RISK  REDUCTION 

associated  with  achieving  reliability  objectives  is  too 

high,  corrective  action  should  be  taken.  The 

recommended  actions  and  the  associated  Risk 

Factor  scores  are  as  shown  in  Table  4-1 . 


TABLE  4-1 

Risk  Reduction 

Risk  Factor 
Score 

Recommendation 

£  0.2w 

None 

>  0.25  -  £  0.4 

Optional  Risk  Reduction  Plan  for 
RefiabiKty 

>  0.4 

Mandatory  Risk  Reduction  Plan 
for  Reliability 

The  quantified  reliability  objectives  described  in 
paragraph  4. 1.4.9  are  the  second  control  metric 
required  for  the  Demonstration  and  Validation 
Phase. 
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Inadequate  Quality 


Inadequate  Definition 
and  Update  ot 
Customer  Requirements 


Faulty  Planning 


Incomplete  or  Inadequat 
Reliability  Analysis  [ 
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Figure  4-9.  Risk  Factor:  Demonstration  and  Validation 

Risk  Factor. P^+C^-P^xC^ 


Probability 
of  Failure 


Crttlara  for  Activity  4.1.1 


Supplier  has  a  consistent,  recognized  and  demonstrated  history  of  high  quality  and  has  completed  a 
CAP  Assessment  or  Malcolm  Baldrige  application  with  a  score  above  500.  if  applicable,  the  supplier 
has  shown  improvement  over  prior  quality  assessments. 


Supplier  has  a  recognized  commitment  to  high  quality,  but  quality  results  are  occasionally  deficient 
Has  completed  a  CAP  Assessment  or  Malcolm  Baldrige  application  with  a  score  above  350.  If 
applicable,  the  supplier  has  shown  improvement  over  prior  quality  assessments. 


Supplier  commitment  to  quality  is  not  completely  evident.  Has  completed  a  CAP  Assessment  or 
Malcolm  Baldrige  applicate  ->  v  fth  a  score  above  250.  If  appl'-able,  the  supplier  has  shown 
improvement  over  prior  quality  assessments. 


Supplier  has  completed  a  CAP  Assessment  or  Malcolm  Baldrige  application  with  a  score  above  150, 
but  has  no  specific  plans  for  improvement. !  a  pit  cable,  the  supplier  has  shown  improvement  over 
prior  quality  assessments. 


Supplier  has  completed  a  CAP  Assessment  or  Malcolm  Baldrige  application  with  a  score  below  150. 


Figure  4-10.  Probability  of  Failure  :  Quality  Evaluation 
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ProbabHRy 
of  Failure 

CrMem  for  Activities  4.1.2  and  4.1.7 

' 

Customer  needs  have  been  Identified  and  prioritized  In  a  systematic  manner,  translated  into  product 
technical  objectives,  and  (lowed- down  to  subsystems  levels.  Trade  study  candMatee  have  been 
identified.  Design  reviews  have  clearly  defined  exit  criteria,  customer  needs  are  reviewed  and  updated, 
compliance  with  customer  needs  are  shown.  Related  research  and  additional  enabling  research 
Identified  and  documented. 

0.3 

Customer  needs  have  been  identified  in  a  systematic  manner,  translated  into  product  technical 
objectives,  and  flowed-down  to  subsystems  levels.  Trade  study  candkiates  have  been  identified. 
Customer  needs  reviewed  and  updated  at  design  reviews.  Compliance  with  customer  needs  are 
shown.  Related  research  and  additional  enabling  research  identified  and  documented. 

0.5 

Customer  needs  have  been  Identified  and  translated  into  product  technical  objectives,  and 
flowed-down  to  subsystems  levels.  Trade  study  candidate  Hst  is  Incomplete.  Design  reviews  show 
compliance  with  customer  needs.  Incomplete  Identification  and  documentation  of  related  research  and 
additional  enabling  research. 

0.7 

Customer  needs  only  partis  ly  identified  with  no  dear  traceabifity  to  product  technical  objectives.  Trade 
study  candidate  list  is  incomplete.  Design  reviews  not  thoroughly  focused  on  customer  needs.  Very 
limited  identification  and  documentation  of  related  research  and  addHional  enabling  research. 

0.9 

Limited  customer/supplier  interchange.  Customer  needs  not  validated  wtth  customer,  design  reviews 
do  not  specifically  address  customer  needs.  No  dear  integration  of  current  research  or  definition  of 
additional  enabling  research. 

CCH-01M  M  mn 


Figure  4-11.  Probability  of  Faiiuro  P^:  Cuatomor  Naada  Interpretation 
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Probability 
of  Failure 

Critiara  for  Activity  4.1.3 

Funding  profiles  are  influenced  by  customer  needs  and  priorities.  All  contractAechnical 
requirements  have  been  addressed.  Risk  reduction  plans  include  reliability  measures  of  merit 

Phase  tasks  are  tailored  to  customer  priorities.  Customer  needs  dearly  transmitted  to  al  personnel. 
Training  needs/plans,  benchmarking  plan  tor  continuous  improvement  identified,  CAE  tools 
available  and  data  bases  integrated.  Allocated  technical  objectives  "flowed  down"  and  clearly 
traceable  to  customer  needs. 

0.3 

Funding  profiles  are  partialy  influenced  by  customer  needs  and  priorities.  Contract/technical 
requirements  have  been  addressed.  Risk  reduction  plans  include  reliability  measures  of  merit 
Customer  needs  not  adequately  transmitted  to  all  personnel.  Training  not  specifically  addressed. 

No  benchmarking  plan.  CAE  tools  list  not  complete.  Allocated  technical  requirements  partially 
traceable  to  customer  needs. 

0.5 

Funding  profiles  not  matched  to  customer  needs  and  priorities.  Contract/technical  requirements 
have  been  addressed.  Risk  reduction  plans  include  reliability  measures  of  merit.  Limited  transmittal 
of  customer  needs  to  all  personnel.  Limited  CAE  tools  defined  and  data  bases  are  not  integrated. 
Allocated  technical  requirements  not  traceable  to  customer  needs. 

0.7 

Funding  inadequate.  Contract/technical  requirements  partially  addressed.  Very  limited  transmittal  of 
customer  needs  to  all  personnel,  very  limited  CAE  tools. 

0.9 

Funding  inadequate.  Contract/technical  requirements  partially  addressed.  No  evidence  of 
transmittal  of  customer  needs  to  personnel.  Extremely  limited  use  of  CAE  tools. 

CC23-0132-6S -O/jm 

Figure  4-12.  Probability  of  Failure  f^3) :  Program  Planning 
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Criteria  for  Activity  4.1.4 

lo  uatormm*  1  otal  scoro,  tsum  tnonnouai  boores  ana  LKvkm  oy  Four  (4) 

|  c IjNnM nVNaMnf  fwwrreiflpf  »w  iPmmgn  umu  1 

0.1 

Quantitative  performance  tavsl  estimate*  baaad  on  comparison  to  currant  cuatomar  experienced  iavala  of  performance  of 
etesting  oomparabi*  equipment.  Clear  engineering  ratio naia  ia  provided  for  ail  expected  improvements.  ferial  definition  of 
doeigrvapplcafion  criteria  oomplato.  Laaaona  laamad  data  ia  aval  labia  (*.g„  derating,  environment*!  sensitivity).  Expected 
opertolng  environment  including  maintenanoa  haa  boon  dalnad  and  tracaabia  to  cuatomar  mission  descnption. 

H 

Quantitative iparformsnca  laval  aatimatoo  baaad  on  comparison  to  currant  Iavala  of  parformanoa  of  existing  comparsbt* 
aquipmanL  On*  or  mors  Kay  parformanoa  paramatora  (a.g.,  Me,  fatigua,  falaa  alarm  rate,  ate.)  is  not  IdanOlad.  Ctoaranginaaiing 
rationale  la  provided  for  al  axpacted  improvements.  Initial  dafMtlon  of  deslgrVappIcation  criteria  complete,  laaaona  laamad  data 
la  aval  labia  (a.g.,  da  rating,  anvironmantal  santetitety).  Expaeted  oparafng  anvironmant  inducing  maintenanoa  has  bean  defined 
and  tracaabia  lo  customer  mission  description. 

H 

Quantttativo  parformanoa  laval  aati mates  baaad  on  comparison  to  currant  levels  of  parformanoa  of  existing  comparable 
equipment.  Kay  parformanoa  parameters  missing  and  engineering  rationale  for  enacted  improvements  is  waste.  Initial  dafotiion 
of  daatgn/appteation  criteria  complete,  laaaona  1 earned  data  ia  aval  labia  (a.g.,  derating,  enteronmerrial  sensitivity).  Expected 
operating  environment  inctoding  maintenanoa  has  bean  dalnad  and  traoaabla  to  cuatomar  mission  description. 

0.7 

Quantttativo  parformanoa  laval  sal  mates  have  Imrtad  rateranoa  to  the  parformanoa  iavala  of  existing  comparable  eautpmant. 
Engineering  raiionaia  tor  expected  improvements  Is  waste.  Limited  definition  of  deaigrVappIcation  coterie  or  lessons  Warned. 
Environment  and  use  data  not  tracaabia  to  the  mission  description. 

0.0 

No  definition  of  existing  comparable  equipment  deaign/applcarion  criteria  or  lessons  foamed.  Environmant  and  use  data  not 
traoaabla  to  too  mission  description 

1  ^  '-'  ■  — - i - __  r,,|.  jinn  ■  I 

1  IrflOW  rWIMaV  Or  runcwons  I 

0.1 

FaUura  modes  a  fleets  analysis  or  similar  analyse*  have  bean  conducted  to  Identify  critical  parameters  or  function*.  Feuti 
detection  and  faut  toforanoa  approaches  systematical  dalnad  and  related  to  critical  paramatora  or  functions.  VariafaMly  at 
critical  parameter*  have  bean  astimaMd. 

0.3 

Failure  modea  effect*  anaiysi*  or  similar  analyse*  have  bean  conducted  to  identify  critical  parameter*  or  functions.  Fault 
detection  and  faut  toforanoa  approaches  systematical  dalnad  and  related  to  critical  par* motor*  or  functions.  Variabiity  of 
critical  parameter*  not  estimated. 

0.5 

FaUura  modes  e fleet*  anaiysi*  or  similar  analyse*  have  bean  conducted  to  Identify  critical  parameters  or  functions.  Fault 
detection  and  faut  toforanoa  approach**  pariUtiy  rotated  to  critical  parameter*  or  unctions  and  not  acoomptehod  concurrently 
with  Mluro  modes  affects  analysis. 

0.7 

Failure  mod**  affects  analysis  or  similar  analyses  have  bean  conducted  to  identify  critical  parameters  or  function*.  No  avhfonoa 
of  Inkaga  between  this  analytes  and  the  definition  of  faut  detection  and  faut  toforanoa  approaches. 

0.9 

FaHura  modes  effect*  analysis  or  similar  analyse*  not  conducted  or  not  Infeed  to  Identifying  critical  parameters  or  functions,  fault 
detection  or  fault  toforanoa  approaches. 

CCS*01S2-4teDf)*m 


Figure  4-13.  Probability  of  Failure  F^4) :  Reliability  Analysis 
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Probability 
of  Failure 

Criteria  for  Activity  4.1.5 

0.1 

Trade  study  effort  comprehensive.  Candidates  selected  based  on  contacts  in  satisfying  customer 
needs.  Alternative  technologies/designs  evaluated  in  a  manner  similar  to  that  done  for  the  baseline. 
Trade  study  parameter  weights  are  traceable  to  priorities  identified  Airing  customer  needs 
interpretation  activity. 

0.3 

Trade  study  effort  comprehensive.  Candkiates  selection  not  completely  traceable  to  conflicts  in 
satisfying  customer  needs.  Alternative  technologies/designs  evaluated  in  a  manner  simlar  to  that 
done  for  the  baseline.  Trade  study  parameter  weights  are  not  completely  traceable  to  priorities 
identified  during  customer  needs  interpretation  activity. 

0.5 

Trade  study  effort  limited.  Candidate  selection  not  completely  justified.  Alternative 
technologies/designs  evaluation  not  as  thorough  as  done  for  the  baseline.  Trade  study  parameter 
weights  are  not  traceable  to  priorities  identified  during  customer  needs  interpretation  acflvNy. 

0.7 

Trade  study  effort  limited.  Candkiate  selection  not  justified.  Limited  effort  in  evaluating  alternative 
technologies/designs.  Trade  study  parameter  weights  not  related  to  customer  prtorfflee. 

0.9 

Trade  study  effort  very  limited.  Results  not  credbie. 

CC23-0132-M-O)mi 

Figure  4-14.  Probability  of  Failura  Ppf  Tradeoff  Analyses 
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1 

ConMqumcM 
of  Failure 
Scot* 

Critter*  for  Mooion  ConooquonoM  (C(1)) 

0.1 

Equipment  Is  not  significant  to  either  mission  completion  or  sortie  generation  capability. 

0.3 

Equipment  provides  moderate  enhancement  to  mission  success. 

0.5 

Equipment  Is  significant  to  mission  success  or  successive  sorties  will  not  be  launched  If 
equipment  Is  Inoperative. 

0.7 

Item  Is  mission  critical.  Failure  will  always  cause  a  mission  abort. 

0.9 

Item  Is  safety  critical. 

Consequences 
of  Failure 
Scon* 

Critters  for  Cost  Consequences  (C^) 

0.1 

Equipment  Is  simple  (£1,000  parts). 

0.3 

Equipment  Is  of  minor  complexity  (1,000  -  2,000  parts). 

0.5 

Equipment  Is  moderately  compietx  (2,000  -  4,000  parts). 

0.7 

Equipment  Is  significantly  complex  (4,000  -  6,000  parts). 

0.9 

Equipment  Is  very  complex  (>6,000  parts). 

CC34-007&-10-0 

Figure  4-15.  Consequence  of  Failure  Criteria 
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These  estimates  must  be  based  on  a  causal  analysis 
of  customer  experience  data  on  similar  equipment 
with  accompanying  engineering  rationale  for  the 
expected  performance  of  the  proposed  system. 

This  method  is  dearly  suitable  for  the  level  of  detail 
available  during  the  Demonstration  and  Validation 
Phase.  The  technique,  if  properly  implemented, 
forces  a  focus  on  the  customer’s  experience  and 
environment  and  preferably  is  accomplished  with 
customer  involvement 

The  process  of  identifying  comparable  existing 
equipment  and  establishing  similarities/dissimilarities 
utilizes  several  of  the  Activities/Inputs/Outputs  that 
are  noted  as  critical  in  the  risk  factor  evaluation.  The 
following  issues  should  be  clearly  identified  for  both 
the  proposed  and  existing  equipments: 

•  Technology  Employed 

•  Complexity 

•  Packaging  Techniques 

•  Design  Margins 

•  Environments  and  Use 

Customer  use  data  must  be  the  basis  for  establishing 
the  performance  of  the  existing  equipment  The  data 
must  indude  measures  of  performance  for  all 
sources  of  faults  (wearout,  damage,  “failures,”  BIT 
performance).  The  fundamental  concern  during  the 
early  stages  of  development  must  be  problem 
identification  and  solution. 

The  final  step  in  the  process  is  the 
identification  of  design  and  technology 
rationale  for  improvements  in  current  levels 
of  reliability  performance.  If,  by  the  end  of 
the  Demonstration  and  Validation  Phase, 
less  than  seventy-five  percent  (75%)  of  the 
difference  between  current  performance  and 
customer  needs  is  supported  by  credible 
engineering  rationale,  corrective  action 
should  be  required  as  part  of  risk  reduction 
plans. 
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5.0  INTRODUCTION 

This  chapter  defines  the  activities  that  should  take 
place  during  the  Engineering  and  Manufacturing 
Development  (EMD)  phase.  Figure  5-1  shows  the 
essential  activities  of  this  phase  and  the  general  time 
phasing  of  these  activities.  Activity  5.1.1,  Quality 
Evaluation,  is  a  continuation  of  the  enterprise-wide 
quality  assessment  initiated  during  the  Concept 
Exploration  phase.  Activity  5.1 .2,  Interpretation  of 
Customer  Needs,  is  the  application  of  Quality 
Function  Deployment  to  systematically  flowdown  the 
customer’s  specification  reliability  requirements  to 
the  lowest  levels  of  design  and  manufacturing.  The 
balance  of  activities,  5.1.3  through  5.1 .9,  define 
detail  design,  manufacturing,  and  test  activity 
focused  on  the  requirements  for  defect  elimination 
tasks.  Specific  descriptions  of  the  phase  activities 
and  their  inputs  and  outputs  are  contained  in 
paragraphs  5.1.1  through  5.1 .9.  In  the  instances 
where  any  of  these  elements  are  common  to 
previously  described  elements,  reference  is  made  to 
the  appropriate  paragraphs.  Paragraph  5.2  defines 
the  appropriate  metrics  required  to  provide  the 
customer  with  assurance  that  the  reliability  process 
is  in  control. 

5.1  SUMMARY  OF  ACTIVITIES 

There  are  three  primary  objectives  of  the  EMD 
phase:  1)  detail  design  implementation  of 
specification  requirements,  2)  development  of  the 
manufacturing  processes  required  to  produce  the 
product,  and  3)  verification,  by  both  analyses  and 
test,  that  the  design  and  manufacturing  processes 
comply  with  customer  requirements. 

The  reliability  process  described  in  paragraphs 

5.1.1  through  5.1.9  support  these  objectives. 
Analytical  verification  of  the  capability  of  the  detail 
design  to  achieve  all  reliability  requirements  is 
accomplished  as  described  in  paragraph  5.1.4.  The 
identification  of  critical  items  is  continued  to 
parameters  of  the  piece  part  level,  and  the 
corresponding  manufacturing  processes  affecting 
these  critical  parameters  are  identified.  The 
capability  indices  for  the  critical  manufacturing 
processes  are  determined  for  the  purpose  of 
eliminating  defects  and  controlling  the 
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manufacturing  process.  Development  tests  are 
conducted  on  risky  design  and  manufacturing 
elements  for  the  purpose  of  validating  and  refining 
analytical  results  and  completing  the 
characterization  of  materials  and  parts.  Verification 
tests  are  conducted  to  confirm  that  the  design  and 
manufacturing  processes  are  capable  of  satisfying 
all  specification  requirements. 

There  are  seven  essential  reliability  activities  during 
the  EMD  phase.  These  are: 

1 )  Quality  Evaluation 

2)  Interpretation  of  Customer  Needs 

3)  Program  Planning 

4)  Detail  Design  Reliability  Analysis 

5)  Trade-Off  Analyses 

6)  Manufacturing  Process  Control 

7)  Development  and  Verification  T ests 

8)  Development  of  Production  Specifications 

9)  Design  Reviews 

Several  of  these  activities  and  their  inputs  and 
outputs  share  common  attributes  with  similar  items 
that  should  have  taken  place  in  both  the  Concept 
Exploration  and  Demonstration  and  Validation 
phases.  The  significant  difference  in  the  EMD 
versions  of  these  activities'  inputs  and  outputs  is  the 
level  of  detail  required  for  the  EMD  phase. 

The  Quality  Evaluation  should  represent  a 
continuation  of  the  drive  to  quantify  the  commitment 
of  the  entire  enterprise  to  total  quality.  If  these 
evaluations  have  been  completed  in  prior 
acquisition  phases,  significant  improvement  should 
be  seen  by  the  EMD  phase.  In  instances  where  this 
is  an  initial  evaluation,  the  results  should  be  used  to 
establish  a  basis  for  specific  improvements. 

The  interpretation  of  customer  needs  activity  is  also 
a  continuation  of  the  process  begun  during  the  Pre- 
Concept  Exploration  Phase.  Firm  specifications  are 
available,  and  Quality  Function  Deployment 
techniques  are  used  to  confirm  the  customer 
priorities  associated  with  reliability  requirements  and 
to  flow  the  requirements  to  the  lowest  levels  of 
design  and  manufacturing  process  detail. 
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Program  planning  uses  activities,  inputs,  and 
outputs  described  in  paragraphs  5.1 .4  through  5.1.9 
to  define  a  tailored  program  plan.  The  plan  is 
adjusted  to  reflect  the  technologies  employed,  the 
severity  of  the  customer’s  environment,  risk  areas, 
and  the  customer’s  priorities  attached  to  reliability 
issues.  The  planning  activity  also  must  develop  test 
plans,  supplier  selection,  and  control  plans,  and 
continue  the  process  of  continuous  improvement 
through  benchmarking  and  training  plans. 

Activity  5.1.4,  Detail  Design  Reliability  Analysis,  is 
one  of  the  central  defect  prevention  activities  of  the 
EMD  phase.  This  activity  uses  detail  design  data, 
design  rules,  and  an  expanded  definition  of  imposed 
stresses  to  create  predictions  of  reliability,  a 
description  of  the  consequences  of  part  parameter 
variability,  a  definition  of  critical  parts 
characteristics  and  related  manufacturing 
processes,  and  a  set  of  design  for  manufacturing 
guidelines.  Inputs  from  development  testing  are 
used  to  refine  the  analytical  results. 

Trade-off  analyses  continue  to  be  used  to  balance 
the  design.  These  analyses  should  focus  on 
simplifying  both  the  design  and  the  manufacturing 
processes. 

The  Manufacturing  Process  Control  activity  is  a 
second  major  activity  directed  at  defect  prevention. 
Its  purpose  is  to  define  the  capability  indices  of  the 
manufacturing  process  and  identify  process  control 
factors  and  the  levels  for  these  factors  associated 
with  minimum  variance.  The  analysis  is  driven  by 
the  definition  of  critical  parts  and  parameters  and  an 
experience  data  base  that  describes  the  current 
capabilities  of  the  manufacturing  process. 

Development  and  Verification  testing  remains  an 
important  part  of  the  EMD  reliability  process.  Both 
development  and  verification  testing  rely  heavily  on 
aggressive  and  comprehensive  failure  reporting 
analyses  and  corrective  action  system.  Testing 
must  be  conducted  using  expected  customer 
environments  representing  realistic  imposed 
stresses.  All  reliability  attributes  must  be  verified, 
including  life,  durability,  BIT  performance,  and 
defect  rates.  Included  in  this  activity  is  the 
development  of  appropriate  Environmental  Stress 
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Screening  (ESS)  programs  for  production 
equipment 

Production  specifications  are  developed  in  activity 
5.1 .8.  These  combine  the  results  of  the  analytical 
activities  5.1 .4  and  5.1 .6  with  the  test  results  of 
activity  5.1 .7  to  create  production  performance 
specifications  and  their  related  manufacturing 
specifications. 

Design  Reviews  (Activity  5.1.9)  continue  to  serve  as 
the  vehicles  for  applying  discipline  and  a  focus  on 
customer  requirements  to  the  reliability  process. 
Design  reviews  must  have  well  defined  entry  and 
exit  criteria  and  include  a  definition  of  applicable 
customer  requirements. 


Input  Activity  Output 


Figure  5-2.  Quality  Baseline 


5.1.1  ACTIVITY  -  QUALITY  EVALUATION  COMMITMENT  TO 

This  pre-contract  award  activity  is  substantially  the  QUALITY 

same  as  defined  in  paragraphs  3.1 .1  and  4.1 .1 .  This 

evaluation  quantitatively  measures  the  supplier’s 

company-wide  commitment  to  quality.  If  these 

evaluations  have  been  previously  conducted, 

substantial  improvements  in  a  supplier’s  score 

should  be  expected  and  demanded.  In  addition, 

suppliers  should  be  expected  to  demonstrate  that 

similar  evaluations  are  conducted  for  subcontractors 

and  suppliers  with  results  used  in  source  selection. 

In  the  conduct  of  this  activity,  it  is  expected  that 
each  major  functional  division  (e.g.,  Manufacturing, 

Engineering,  etc.)  within  an  organization  will  assess 
their  activities  in  light  of  the  criteria  contained  in 
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Volume  II  of  this  guide.  At  the  organization  level,  the 
inputs  from  the  functional  level  are  to  be  aggregated 
and  compiled  into  a  single  report  for  submittal  to  the 
customer. 

5.1. 1.1  INPUT  -  EVALUATION 
INSTRUCTIONS 

The  customer  includes  in  his  request  for  proposal  a 
complete  set  of  instructions  for  conducting  the 
quality  evaluation.  These  include  directions  in 
Executive  Summaries,  detail  instructions  in  the 
Instructions  to  Offerors  (Evaluation  Criteria, 
Reporting  Requirements),  and  the  relationship  of  the 
evaluation  to  source  selection  award  factors.  An 
example  of  instructions  is  provided  following 
paragraph  3.1  .1  .3.  Report  format  requirements  for  a 
Malcolm  Baldrige  Award  Application  are 
recommended. 

5.1. 1.2  INPUT  -  EVALUATION  CRITERIA 

The  customer  supplies  the  criteria  to  be  used  for  the 
quality  evaluation.  Volume  II  of  this  guide  provides 
a  set  of  recommended  criteria  patterned  after  the 
Malcolm  Baldrige  Award  Criteria.  Suggested 
scoring  is  also  provided. 

5.1. 1.3  OUTPUT  -  QUALITY  EVALUATION 
REPORT 

The  supplier  provides  a  quality  evaluation  report  in 
accordance  with  customer  reporting  requirements. 
This  report  is  to  be  used  by  both  the  customer  and 
the  supplier.  The  customer  will  score  the  results  and 
use  this  in  source  selection.  The  supplier  should 
similarly  score  his  results  and  use  this  evaluation  to 
define  weaknesses  and  plan  the  required 
improvements. 
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5.1.2  ACTIVITY  -  INTERPRET  CUSTOMER 
NEEDS 

During  the  EMD  phase,  firm  performance  and 
verification  specifications  are  available.  However, 
a  systematic  and  comprehensive  evaluation  of 
customer  needs  and  priorities  is  still  a  mandatory 
activity,  with  Quality  Function  Deployment  QFD  as 
the  recommended  tool.  Customer  requirements  and 
priorities  must  be  translated  into  product  technical 
requirements  and  flowed  down  to  the  component 
and  manufacturing  process  level  using  QFD 
techniques.Direct  interchange  between  customer 
and  supplier  are  required  for  the  proper 
implementation  of  this  activity.  There  will  be  several 
iterations  of  the  QFD  as  the  requirements  flow  from 
the  system  to  subsystem  to  assembly  to  lower  levels 
of  design  and  manufacturing  detail. 

5.1 .2.1  INPUT  -  CUSTOMER 
REQUIREMENTS 

Customer  needs  and  objectives  should  be  explicitly 
defined  in  the  contract  technical  documentation, 
such  as  Statements  of  Work  and  specifications. 
However,  all  contract  documentation  must  be 
reviewed  for  descriptions  of  customer  requirements. 
Additionally,  the  supplier  must  plan  for  early,  direct, 
and  frequent  customer  contact  to  validate  and  refine 
the  documented  needs.  These  contacts  must  focus 
on  a  multidisciplinary  definition  of  customer  needs. 

5.1. 2.2  INPUT  -  EXPERIENCE  DATA 

The  supplier  must  ensure  that  customer  documented 
requirements  are  subject  to  review  by  experts  in  all 
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areas  so  that  all  requirements  are  identified,  with 
correct  priorities,  property  translated  into  design  and 
manufacturing  requirements,  and  flowed  down.  The 
supplier  must  also  ensure  that  QFDs  completed 
during  the  prior  acquisition  phase  are  available  for 
application  to  the  EMD  contract.  The  experience 
data  base  should  include  a  thorough  and 
documented  understanding  of  good  and  bad 
customer  experience  with  current  products  as  a 
preferred  method  for  aiding  in  the  identification  of 
requirements. 

5.1. 2.3  OUTPUT  -  PRODUCT  DESIGN  AND 
MANUFACTURING  REQUIREMENTS 

The  supplier  must  provide  a  definition  of  specific 
technical  attributes  and  parameters  that  will  satisfy 
all  customer  requirements.  This  definition  includes  - 
quantitative  values  for  each  of  the  attributes  and 
parameters.  The  customer  needs  must  be  given 
weighting  factors  which  represent  their  priority 
levels.  These  factors  will  be  used  to  define  the 
relative  importance  of  each  of  the  technical 
attributes  and  parameters.  It  is  critical  that  the 
interaction  between  reliability  technical  attributes 
and  the  other  technical  attributes  be  identified  for  the 
purpose  of  identifying  trade  study  candidates. 

The  technical  requirements  should  be  flowed  down  COMPREHENSIVE 

by  QFD  procedures  to  the  component  and  RELIABILITY 

manufacturing  process  level  of  indenture.  This  REQUIREMENTS 

flowdown  will  usually  require  several  iterations 

during  the  EMD  phase.  The  flowdown  must  maintain 

traceability  to  customer  requirements  and,  more 

importantly,  the  customer’s  priorities  for  these 

requirements.  In  order  to  retain  focus  on  an 

integrated  set  of  reliability  requirements,  it  is 

recommended  that  all  the  product  reliability 

attributes  be  grouped  together  during  the  QFD 

development.  These  attributes  include  service  life, 

durability,  defect  rates,  and  BIT  performance 

requirements  (e.g.,  False  Alarm  rate,  Fault  Detection 

Probability,  Could  Not  Duplicate  (CND)  rate,  etc.). 

5.1. 2.4  OUTPUT  -  TRADE  STUDY 
CANDIDATES 

The  supplier  must  identify  candidates  for  trade 
studies  to  be  performed  during  the  EMD  Phase.  The 
Quality  Function  Deployment  techniques  can  assist 
in  defining  these  candidates  by  evaluating  the 
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interactions  among  technical  attributes.  Trade-off 
analyses  are  used  to  optimize  a  technical  attribute  or 
parameter  that  interacts  with  other  technical 
attributes  or  parameters.  For  example,  radar 
detection  range  is  a  function  of  power,  weight 
volume,  receiver  sensitivity,  reliability,  and  cost. 

Trade  studies  are  also  used  to  select  between 
alternative  design  and  manufacturing  approaches. 

5.1 .2.5  OUTPUT  -  EMD  RISK  REDUCTION  CONTINUOUS  RISK 

PLAN  REDUCTION 

The  baseline  reliability  risk  reduction  plan  is  that 
prepared  for  Output  4.1 .6.6  at  the  conclusion  of  the 
Demonstration  and  Validation  phase.  The  risk 
issues  addressed  here  are  those  not  satisfactorily 
closed  during  the  Demonstration  and  Validation 
phase  or  those  that  are  determined  by  the  Control 
and  Audit  requirements  of  paragraph  4.2.  If  a 
supplier  has  not  participated  in  the  Demonstration 
and  Validation  phase,  an  initial  risk  reduction  plan 
must  be  prepared.  The  plan  must  include  the  criteria 
for  selection  of  risk  issues.  The  output  of  the  QFD 
should  be  used  to  establish  a  list  of  technical 
requirements  rank-ordered  based  on  customer 
priorities.  This  list  then  serves  as  a  baseline  for 
establishing  priorities  for  the  risk  issues.  The  risk 
reduction  plan  must  describe  the  criteria  used  to 
select  the  risk  issues.  Criteria  can  consider 
elements  such  as  new  technology  parts  and 
materials,  prior  negative  reliability  history,  life  limited 
items,  parts  and  materials  which  are 
uncharacterized  relative  to  the  expected  use 
environments  or  manufacturing  technologies  whose 
capability  indices  are  low  or  have  not  been 
established.  The  plan  must  also  describe  the 
analyses  and  tests  required  to  establish  either  defect 
prevention  design  margins  or  manufacturing  process 
controls.  This  includes  the  identification  of 
measures  of  merit  that  will  be  tracked  and  thresholds 
that  identify  success.  Finally,  the  plan  must  identify 
fall  back  positions  that  will  be  implemented  if  risks 
cannot  be  resolved  and  the  milestones  at  which 
those  decisions  will  be  made. 
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5.1.3  ACTIVITY  -  PROGRAM  PLANNING  MULTIDISCIPLINE 

Much  of  this  activity  is  substantially  the  same  as  PLANNING 

described  in  paragraphs  3.1.3  and  4.1 .3.  The  inputs 

and  outputs  of  the  remaining  phase  activities  (5.1.4 

through  5.1.9)  serve  to  identify  the  baseline  tasks  to 

be  accomplished  during  the  EMD  phase.  The 

tailoring  is  driven  by  the  inputs  to  this  activity  which 

include  customer  priorities,  trade-off  candidates, 

EMD  risk  reduction  plans,  and  contract 
requirements.  The  resultant  plans  must  identify 
specific  responsibilities  for  inputs  and  outputs, 
schedules  for  these  tasks,  and  data  required.  The 
plan  should  be  coauthored  by  all  the  functional 
specialties  with  principal  involvement  in  defect 
prevention  activities  (Figure  1-19). 

5.1 .3.1  INPUT  •  FUNDING  PROFILES 

The  allocation  of  funds  must  be  interactive,  flexible, 
and  demonstrably  shown  to  be  related  to  the 
customer  priorities  identified  as  a  result  of  Activity 
5.1 .2.  The  description  of  this  input  is  otherwise  the 
same  as  provided  in  paragraphs  3.1. 3.1  and  4.1 .3.1. 
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5.1 .3.2  INPUT  -  EMD  RISK  REDUCTION 
PLAN 

Output  5. 1.2.5  serves  as  an  input  to  the  Planning 
Activity.  This  input  establishes  the  reliability  risk 
reduction  analyses  and  test  tasks  including  parts 
and  materials  environmental  characterization  test 
requirements. 

5.1 .3.3  INPUT  -  CONTRACT 
REQUIREMENTS 

The  description  of  this  input  is  the  same  as  provided 
in  paragraph  3. 1.3.2 

5.1 .3.4  INPUT  -  PRODUCT  DESIGN  AND 
MANUFACTURING  REQUIREMENTS 

Output  5.1 .2.3  serves  as  an  input  to  the  Planning 
Activity.  Quantified  and  prioritized  reliability 
requirements  establish  the  focus  and  direction  for 
program  planning  outputs. 

5.1 .3.5  INPUT  -  TRADE  STUDY 
CANDIDATES 

Output  5.1 .2.4  serves  as  an  input  to  the  Program 
Planning  Activity.  Additional  trade  study  candidates 
may  have  been  developed  as  a  result  of  risk 
reduction  plans,  customer  requests,  or  supplier 
experience.  The  candidate  list  should  define  the 
specific  measures  of  merit  being  used  to  evaluate 
trade  study  alternatives.  Any  weighting  factors 
applied  to  the  trade  study  measures  of  merit  should 
be  shown  in  the  candidate  list  and  should  be 
traceable  to  customer  priorities.  Reliability  attributes 
and/or  parameters  should  be  considered  for  all  trade 
studies  as  measures  of  merit. 

5.1 .3.6  OUTPUT  -  TAILORED  PROGRAM 
PLAN 

The  supplier  should  prepare  a  program  plan  for  the 
EMD  phase  using  the  activities,  inputs,  and  outputs 
of  paragraph  5.1 .4  through  5.1.9  as  the  baseline  for 
the  tasks  to  be  performed.  Plans  must  be  tailored 
based  on  the  complexity  of  the  equipment,  the 
expected  customer  use  environment,  and  contract 
requirements.  The  task  descriptions  may  use 
organization-wide  process  descriptions  if  they  are 
comprehensive  and  cover  the  intent  of  the  activities, 
inputs,  and  outputs  described  in  paragraphs  5.1.4 
through  5.1.9.  Plans  must  identify  task  acceptance 
(completion)  criteria,  responsibility  for  all  tasks,  and 
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schedules  for  both  task  input  and  output  products. 

5.1. 3.7  OUTPUT  -  CAE  TOOLS 

The  description  of  this  output  is  the  same  as 
provided  in  paragraphs  3. 1.3.6  and  4.1.3.11. 

5.1. 3.8  OUTPUT  -  REQUIREMENTS 
FLOWDOWN 

If  not  accomplished  as  part  of  Activity  5.1.2,  the 
quantitative  and  qualitative  (e.g.,  fault  tolerance) 
requirements  identified  in  Output  5. 1.2.3  must  be 
allocated  to  all  necessary  levels  of  indenture  and  to 
all  procured  assemblies.  The  allocation  process 
must  retain  the  definition  of  customer  priorities 
among  reliability  requirements  and  between 
reliability  requirements  and  other  design  and 
manufacturing  requirements.  The  allocation  process 
must  account  for  the  historical  user  experience  with 
like  and  similar  equipment. 

5.1. 3.9  OUTPUT  -  SUBCONTRACTOR  CONTROL  SUPPLIERS 

SELECTION  AND  CONTROL  PLAN 

This  plan  describes  the  supplier  and  subcontractor 
actions  needed  to  ensure  that  all  EMD  phase 
requirements  are  assigned  to  suppliers  and 
subcontractors.  This  plan  includes  a  definition  of 
supplier  or  subcontractor  selection  criteria  including 
the  application  of  the  Quality  Evaluation  activity  of 
paragraph  5.1 .1 .  This  task  may  be  accomplished  as 
part  of  any  existing  enterprise-wide  supplier 
certification  program.  The  plan  should  specifically 
address  the  selection  and  control  of  high  quality 
parts  and  materials  with  a  focus  on  determining  the 
process  control  exercised  by  parts  and  materials 
suppliers.  The  plan  should  also  describe  the 
procedures  for  ensuring  the  timely  conduct  of  a 
Quality  Function  Deployment  between  the 
contractor  and  his  suppliers  or  subcontractors. 

5.1.3.10  OUTPUT  -  INTEGRATED  TEST  PERFORMANCE 

PLAN  VERIFICATION 

This  output  is  a  comprehensive  definition  of  all 
reliability  related  development  and  verification 
testing  and  the  failure  reporting  analyses  and 
corrective  action  system  that  will  be  employed 
during  these  tests.  The  purpose  of  the  test  plan  is  to 
ensure  that  all  reliability  attributes  are  validated,  that 
testing  is  not  duplicated,  that  the  test  schedule  and 
resources  are  adequate,  and  that  test  faults  are 
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corrected  by  design  or  manufacturing  process 
changes.  The  testing  that  must  be  defined  includes: 

•  Risk  reduction  tests,  such  as  parts  and 
materials  properties  characterization  tests, 
parameter  variability  determination, 
subassembly  performance  tests,  and 
manufacturing  process  development  tests. 

•  Performance  verification  testing  at 
environmental  extremes. 

•  Life  and  durability  verification  testing. 

•  Reliability  growth  or  verification  testing. 

•  Built-In-Test  development  and  verification  test. 

•  Environmental  Stress  Screening  (ESS)  and 
equipment  acceptance  tests. 

Each  test  from  the  above  categories  must  be 
identified,  scheduled,  and  have  the  following  issues 
addressed: 

•  Definition  of  test  objectives  and  the 
measurable  criteria  for  success. 

•  Test  sample  requirements. 

•  Environmental  and  operating  conditions. 

•  Test  duration  and  test  facilities  required. 

•  Responsibility  for  detail  test  procedures  and 
reporting. 

•  Failure  reporting  analysis  and  corrective 
actio*  ,  requirements,  including  responsibility 
for  all  elements  of  this  process  and  the 
corrective  action  close-out  status  tracking 
system. 

Several  key  issues  must  be  addressed  in  the  test  REALISTIC  TESTING 

plan.  The  applied  environments  must  represent 

customer  use  environments  (thermal,  electrical, 

mechanical)  and  customer  operation  including 

maintenance.  Life  testing  must  be  long  enough  and 

of  sufficient  severity  to  verify  fatigue  resistance  and 
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durability.  AH  failures  must  be  subjected  to  a 
thorough  causal  analysis,  such  that  broadly 
applicable  design  and  manufacturing  process 
changes  can  be  implemented. 

5.1.3.11  OUTPUT  •  BENCHMARKING  AND 
TRAINING  PLAN 

The  benchmarking  section  of  this  plan  should 
address  any  weaknesses  in  the  supplier’s  reliability 
processes  uncovered  by  the  quality  evaluation  of 
paragraph  5.1 .1 .  The  plan  should  address  the 
benchmarking  issues  described  in  paragraph  1.1.2 
and  Figure  1-7.  The  benchmarking  should  be 
directed  at  improving  the  reliability  processes 
directly  affecting  the  EMO  phase.  Candidates  for 
consideration,  in  addition  to  those  listed  in 
paragraph  4.1.3.10,  are: 

•  Design/Electronic  Circuit  Simulation 
Techniques 

•  Design  and  Manufacturing  Variability  Control 

•  Physics  of  Failure  Analyses. 

The  training  position  of  the  plan  must  address  the 
capability  of  program  personnel  to  accomplish  the 
activities,  inputs,  and  outputs  defined  in  paragraphs 
5.1.4  through  5.1.9  (e.g.,  Design  for  Manufacturing, 
Variability  Control,  etc.),  and  identify  training  needs 
and  the  resources  to  satisfy  those  needs.  The 
training  plan  must  also  describe  the  methods  for 
disseminating  information  relative  to  reliability 
requirements  to  all  affected  program  personnel. 
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Input  Activity  Output 


Figure  5-5.  Detail  Deeign  Analyeie 


5.1.4  ACTIVITY  -  DETAIL  DESIGN 
RELIABILITY  ANALYSIS 

This  is  one  of  the  key  activities  of  the  EMD  phase. 
There  a^e  ’four  principal  subactivities.  The  definition 
of  design  rules,  begun  during  the  Demonstration  and 
Validation  phase,  is  continued  and  refined  based  on 
detail  stress  analyses  and  results  from  development 
tests.  The  stress  analyses  and  predictions  also  yield 
quantitative  estimates  of  reliability  attributes  (life  and 
defect  rate  including  BIT  and  manufacturing 
defects).  Critical  product  characteristics  are  defined 
down  to  the  component  and  parameter  levels. 

These  in  turn  drive  variability  control  analyses  and 
the  identification  of  critical  manufacturing  processes. 
Lastly,  design  for  manufacturing  guidelines  are 
defined  in  order  to  simplify  both  the  design  and  the 
manufacturing  process.  Much  of  this  activity  builds 
on  the  results  of  the  Demonstration  and  Validation 
Activity,  4.1.4.  If  the  equipment  being  developed 
has  not  been  subjected  to  a  formal  Demonstration 
and  Validation  phase,  several  of  the  Inputs  and 
Outputs  described  in  Chapter  4  must  be 
accomplished  as  part  of  this  phase. 

5.1.4. 1  INPUT  -  PRODUCT  DESIGN  DATA 

Complete  electrical,  mechanical,  and  thermal 
engineering  design  descriptions  are  required  for  this 
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analysis.  These  data  include: 

•  Function  block  diagrams,  functional 
descriptions,  circuit  schematics,  circuit  board 
layout  drawings,  packaging  and  assembly 
drawings,  parts  lists,  parts  performance  data, 
thermal  models,  and  properties  related  to 
dynamic  analyses  of  the  equipment. 

•  Detail  parts  and  material  characterization  data 
(response  to  environmental  stress). 

•  Parts  and  materials  parameters  variability  data. 

These  data  are  vital  to  variability  control.  The 
distribution  of  critical  parts  and  materials 
parameters  must  be  obtained  from 
suppliers/manufacturers  or  by  test.  This  issue 
is  also  an  important  topic  for  inclusion  in 
parts/materials  specifications  and  control 
plans. 

•  Parts  and  Materials  quality  control  parameters, 
such  as  defect  per  million  requirements, 
screening  and  test  requirements,  and 
manufacturing  process  control  elements. 

•  Manufacturing  and  Assembly  drawings  and 
the  initial  description  of  all  manufacturing 
process  steps. 

If  the  equipment  has  not  been  subject  to  a 
Demonstration  and  Validation  phase,  the  data 
described  in  the  last  item  of  paragraph  4.1 .4.1  is  also 
applicable. 

5.1 .4.2  INPUT  -  DESIGN  CRITERIA  DEFECT 

This  input  represents  a  continuing  refinement  of  the 
data  described  in  paragraph  4.1. 4.7,  Design  Criteria 
Report.  These  data  are  a  comprehensive  set  of 
design  rules  including  parts  and  materials 
application  guides,  lessons  learned,  derating 
requirements,  thermal  design  guides,  BIT/testability 
design  guides,  and  manufacturing  and  assembly 
design  rules.  Special  attention  should  be  paid  to 
any  new  or  untried  technologies  and  components 
having  a  poor  record  of  reliability  performance. 


PREVENTION 
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5.1. 4.3  INPUT  -  REQUIREMENTS 
FLOWDOWN 

Output  5. 1.3.8  serves  as  an  input  to  this  activity. 
These  data  are  the  customer  quantitative  reliability 
requirements. 

5.1 .4.4  INPUT  -  ENVIRONMENT  AND  USE 
PROFILES 

This  input  is  a  refinement  of  the  data  described  in 
paragraph  4. 1.4.4.  The  supplier  is  responsible  for 
defining  local  life  cycle  stress  profiles  that  result  from 
external  environments,  including  power  and  cooling 
conditions,  and  equipment  operation  during  all  use 
including  all  forms  of  maintenance.  The  data  must 
show  both  maximum  expected  stress  levels  and 
durations  and  total  life  cycle  exposure  profiles 
based  on  service  life,  basing  locations,  and 
types/durations  of  missions  and  other 
operations/use.  All  damaging  environments  should 
be  defined  in  the  manner  described  above. 

5.1. 4.5  INPUT  -  DEVELOPMENT  TEST 
RESULTS 

Development  tests  include  risk  reduction  tests,  such 
as  parts  and  materials  characterization  tests  and 
element/subassembly  performance  verification  tests. 
These  tests  are  identified  in  paragraphs  5. 1.2.5  and 
5.1.3.10.  All  development  testing  should  include 
data  from  which  conclusions  regarding  reliability 
can  be  drawn.  A  key  element  in  development  test 
results  is  a  comprehensive  failure  reporting 
analyses  and  corrective  action  system. 

5.1. 4.6  OUTPUT  -  STRESS  ANALYSIS  AND 
PREDICTIONS 

This  output  represents  a  complete  evaluation  of 
imposed  stresses  and  the  capability  of  the  design  to 
withstand  these  stresses.  The  analysis  consists  of 
the  following  elements: 

•  Electrical  stress  margins,  including 
performance  under  transient  conditions. 

•  Mechanical  stress  margins  to  include 
vibration,  g-loading,  and  thermal  stresses. 
Analyses  should  include  solder  joint  fatigue 
and  other  deterioration  mechanisms. 


TEST  RESULTS 


PHYSICS  OF  FAILURE 
ANALYSIS 
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•  Testability  analyses  proving  the  accuracy  of 
BIT  design. 

•  Quantitative  estimates  of  maintenance 
frequency  to  include  both  unscheduled 
maintenance  resulting  from  all  causes  and  any 
proposed  scheduled  maintenance.  These 
predictions  must  exceed  customer 
requirements  and  be  based  on  compliance 
with  the  design  criteria  of  paragraph  5.1 .4.2.  At 
times  MIL-HDBK-217  predictions  may  not  be 
complete  and  may  require  supplements  such 
as  physics  of  failure  and  fatigue  analysis. 

Prediction  results  should  also  be  compared  to 
customer  experience  with  similar  equipment 
and  differences  justified  by  tee  stress  analyses 
cited  above,  the  design  criteria  defined  in 
paragraph  5.1 .4.2,  or  the  results  of  the 
development  test  defined  in  paragraph  5.1 .4.5. 

Predictions  should  be  iterated  throughout  the 
EMD  phase.  Improvements  over  initial 
estimates  must  be  shown  and  must  be  based 
on  design  simplification,  increased  stress 
margins  (exclusive  of  improved  cooling),  or 
changes  in  the  implementation  of  the  BIT 
design. 

5.1. 4.7  OUTPUT  -  VARIABILITY  ANALYSIS  CONTROL  VARIANCE 

In  addition  to  estimates  of  maintenance  frequency,  a 
comprehensive  variability  analysis  and  control 
program  should  be  implemented  for  both  the  design 
and  manufacturing  processes.  The  program  should 
be  based  on  the  outputs  of  paragraph  5. 1.4.8  which 
defines  critical  product  characteristics  and  the 
corresponding  critical  manufacturing  processes. 

Parametric  and  geometric  variability  should  be 
established  from  test  data  and  should  be  included  as 
a  parts  and  materials  control  requirement.  The 
objective  of  the  variability  analyses  should  be  the 
demonstration  that  the  nominal  values  of  critical 
parameters  are  six  sigma  from  specification  limits. 

The  variability  analyses,  including  worst  case 
evaluations  should  be  used  to  verify  the  margins 
and  sensitivity  of  the  equipment  BIT  design. 

Variability  should  also  be  included  in  stress  and 
fatigue  analyses  to  ensure  the  adequacy  and 
accuracy  of  these  analyses.  The  variability 
analyses  should  make  maximum  use  of  Computer 
Aided  Engineering  (CAE)  tools  and  computer 
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simulation  to  evaluate  the  consequences  of  parts 
and  materials  variability.  This  output  should  define 
the  tools  that  will  be  employed  and  the  specific 
methodology,  inputs,  and  assumptions  necessary 
for  the  operation  of  these  tools. 

5.1. 4.8  OUTPUT  -  CRITICAL  PRODUCT  CRITICAL  ITEMS 

CHARACTERISTICS  AND  MANUFACTURING 

PROCESSES 

This  output  is  the  primary  driving  force  behind  three 
key  issues:  1)  variability  control,  BIT  design,  and 
fault  tolerance  design.  There  are  three  sources  for 
the  definition  of  critical  product  characteristics:  1) 
the  QFD  interpretation  o  customer  requirements,  2) 

Failure  Modes  Effects  and  Criticality  Analyses,  and 
3)  engineering  experience.  Critical  functions  and 
critical  signals  are  initially  identified  and,  as  the 
design  is  detailed,  critical  assemblies,  parts,  and 
parameters  of  the  design  are  identified  by  the  three 
methods  described  above.  These  evaluations  must 
take  place  as  the  design  is  being  developed. 

The  design  of  BIT  functions,  fault  tolerance  features, 
and  the  identification  of  critical  items  for  variability 
control  must  be  shown  to  be  traceable  to  the 
analyses  defined  by  this  output.  These  results  must 
also  be  used  to  identify  the  manufacturing  processes 
affecting  the  critical  product  characteristics  in  order 
to  establish  priorities  for  the  control  of  manufacturing 
processes.  Customer  inputs  are  necessary  to 
properly  define  all  critical  product  characteristics. 

5.1. 4.9  OUTPUT  -  DESIGN  FOR 
MANUFACTURING  GUIDELINES 

A  central  element  is  the  achievement  of  both 
quantitative  reliability  requirements  and  control  of 
variability  in  an  explicit  set  of  design  for 
manufacturing  guidelines.  These  should  emphasize 
simplicity,  commonality,  and  the  use  of  standardized 
manufacturing  processes.  These  guides  must  also 
define  any  special  handling  or  processing 
constraints. 


5-19 


CHAPTER  5 

ENGINEERING  AND  MANUFACTURING  DEVELOPMENT  PHASE 


5.1.5  ACTIVITY  -  TRADE-OFF  ANALYSIS 

This  activity  is  substantially  the  same  as  defined  in 
paragraphs  3.1 .5  and  4.1.5.  If  equipment  has  not 
been  the  subject  of  concept  exploration  or 
demonstration  and  validation  phases,  the  initial 
trade-off  analyses  must  be  directed  at  the  selection 
of  technologies  to  be  employed  and  the  refinement 
of  system  level  parameters.  The  remainder  of  the 
trade-off  analyses  are  directed  at  balancing  detail 
design  implementation  issues  with  emphasis  on  the 
selection  of  specific  parts,  materials,  and 
manufacturing  processes. 

5.1 .5.1  INPUT  TRADE  STUDY  CANDIDATES 

Output  5.1 .2.4  serves  as  an  input  to  the  trade  study 
activity.  This  input  is  a  comprehensive  list  of  the 
trade  studies  to  be  conducted,  the  objectives  of 
these  trades,  the  parameters  to  be  evaluated,  and 
the  responsibilities  for  the  conduct  of  these 
analyses. 

5.1. 5.2  INPUT  -  REQUIREMENTS 
FLOWDOWN 

Output  5.1 .3.8  serves  as  an  input  to  the  trade-off 
analysis  activity.  This  output  defines  the 
quantitative  customer  reliability  requirements  that 
must  be  met. 


5.1. 5.3  INPUT  -  DESIGN  AND 
MANUFACTURING  ALTERNATIVES 

Each  alternative  design  implementation  or 
manufacturing  process  for  each  trade-off  analysis 
should  be  described  as  defined  in  paragraph 
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5.I.4.I. 

5.1. 5.4  OUTPUT  -  TRADE  STUDY  REPORTS 

The  description  of  this  output  is  the  same  as 
contained  in  paragraphs  4. 1.5.4  and  3. 1.5.4.  Trade 
study  results  are  subject  to  customer  review  and 
approval. 


5.1.6  ACTIVITY  -  MANUFACTURING  CONTROL  THE 

PROCESS  CONTROL  MANUFACTURING 

This  activity  defines  the  requirements  necessary  for  PROCESS 
the  control  of  the  manufacturing  process  and  the 
prevention  of  defects  introduced  by  manufacturing. 

This  is  also  the  second  element  of  variability  control. 

Design  activity  focuses  on  the  relationship  between 
parameter  variance  and  allowable  tolerances;  this 
activity  defines  the  steps  necessary  to  achieve  the 
required  variance  limits.  The  activity  includes  the 
flow-through  of  requirements  to  parts  and  materials 
suppliers.  The  outputs  of  this  activity  define 
manufacturing  process  capability  indices,  the 
factors  which  control  the  manufacturing  process 
results,  and  the  control  limits  imposed  on  these 
variables. 
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5.1 .6.1  INPUT  -  DESIGN  FOR 
MANUFACTURING  GUIDELINES 

Output  5. 1.4.9  serves  as  an  input  to  this  activity. 
These  guidelines  must  be  iteratively  developed  with 
design  engineering  and  must  reflect  the  capabilities 
and  limitations  of  the  manufacturing  process. 

5.1 .6.2  INPUT  -  CRITICAL  PRODUCT 
CHARACTERISTICS  AND  MANUFACTURING 
PROCESSES 

Output  5.1 .4.8  serves  as  an  input  to  this  activity. 

The  control  of  manufacturing  processes  must  be 
clearly  linked  to,  and  priorities  established  by,  the 
definition  of  critical  product  characteristics. 

5.1 .6.3  INPUT  -  MANUFACTURING 
PROCESS  DESCRIPTION 

This  input  is  a  complete  description  of  the 
manufacturing  process  and  the  capability  index 
associated  with  each  step  in  the  process.  These 
descriptions  must  be  provided  for  those  processes, 
including  suppliers  processes,  which  affect  critical 
product  characteristics.  This  input  should  highlight 
process  steps  for  which  the  capability  index  has  not 
been  defined. 

5.1 .6.4  INPUT  -  MANUFACTURING 
EXPERIENCE  DATA 

This  input  is  the  manufacturing  process  equivalent  to 
the  design  input  described  in  paragraph  5. 1.4.2. 

The  data  includes  defect  rates  for  existing 
manufacturing  process  steps,  corrective  actions  to 
be  implemented  as  part  of  the  EMD  phase,  lessons 
learned  for  inclusion  in  design  for  manufacturing 
guidelines,  and  process  control  variables  and 
control  limits  for  existing  manufacturing  process 
steps. 

5.1. 6.5  OUTPUT  -  MANUFACTURING 
PROCESS  FMECA 

A  failure  modes  effects  and  criticality  analysis 
should  be  conducted  on  the  equipment  required  to 
implement  the  most  important  manufacturing 
processing  steps.  These  analyses,  or  similar 
analyses,  represent  a  continuing  progression  in  the 
identification  of  critical  elements.  These  analyses 
can  be  used  to  ensure  the  reliability  of 
manufacturing  equipment,  define  fault  indication  and 
monitoring  requirements,  and  provide  the  basis  for 


MANUFACTURING 

EQUIPMENT 

CRITICALITY 
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maintenance  practices  and  manufacturing 
workaround  plans. 

5.1 .6.6  OUTPUT  -  MANUFACTURING 
PROCESS  CONTROL  PLAN 

This  plan  applies  to  all  critical  manufacturing 
processes.  It  defines  the  capability  indices  for  these 
processes,  the  control  variables  for  these  processes 
and  their  limits,  and  the  description  of  process 
control  elements  including  Statistical  Process 
Control  (SPC)  implementation  requirements.  The 
plan  should  define  the  methods,  tests,  and  analyses 
that  will  be  implemented  for  manufacturing  process 
steps  that  are  not  completely  characterized.  The 
plan  should  also  define  the  criteria  and  capability  for 
conducting  Design  of  Experiments  for  the  purpose  of 
reducing  process  variability.  This  plan  should 
include  flow  through  requirements  to  ensure  the 
control  of  purchased  parts  and  materials. 


5.1.7  ACTIVITY  -  DEVELOPMENT  AND 
VERIFICATION  TESTS 

This  activity  includes  the  complete  spectrum  of  tests 
required  to  develop  and  verify  product  performance 
requirements.  All  testing  should  be  expected  to 
yield  data  required  for  defect  prevention  or 
elimination. 

5.1.7.1  INPUT  -  INTEGRATED  TEST  PLAN  COMPREHENSIVE 

Output  5.1 .3.1 0  serves  as  an  input  to  this  activity.  TESTING 

This  input  defines  all  tests,  test  conditions,  test 
objectives,  and  failure  reporting  analyses  and 
corrective  action  system  (FRACAS)  requirements. 

Testing  should  include  performance  under  expected 
environmental  extremes,  life  testing  for  fatigue  and 
wearout  mechanisms  using  representative  models  of 
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the  operational  equipment  long  term  mission 
environment  tests  to  validate  requirements,  such  as 
Mean  Time  Between  Maintenance,  and  all  parts, 
materials,  and  subassembly  development  tests.  A 
comprehensive  FRACAS  that  results  in  design  and 
manufacturing  process  changes  is  an  essential 
element  of  the  test  plan. 

5.1. 7.2  INPUT  -  ENVIRONMENT  AND  USE 
PROFILES 

Input  5.1. 4.4  also  serves  as  an  input  to  this  activity. 
All  testing,  especially  life  and  long  term  performance 
testing,  should  be  conducted  under  conditions 
derived  from,  and  traceable  to,  customer  use. 

5.1 .7.3  OUTPUT  -  TEST  REPORTS  AND 
DESIGN  CHANGES 

All  test  results  and  all  corrective  actions  resulting 
from  these  tests  should  be  reported  for  customer 
approval.  The  key  to  this  output  is  the  definition  of 
the  procedures  for  ensuring  that  changes  are 
incorporated  in  the  equipment  design  or 
manufacturing  processes  and  the  criteria  for 
verifying  the  effectiveness  of  these  changes. 

This  output  also  defines  the  stress  screening  that  will 
be  applied  to  production  equipment.  This  includes 
both  subassembly  and  system  level  screening.  The 
stress  levels  and  duration  of  these  tests  should  be 
based  on  the  results  of  development  test  and  long 
term  performance  tests.  A  sufficient  body  of 
knowledge  exists  in  the  literature  to  fully  and 
adequately  define  these  tests. 
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5.1.8  ACTIVITY  -  PRODUCTION 
SPECIFICATIONS 

Production  specifications  that  include  both  product 
performance  and  verification  and  manufacturing 
requirements  are  developed  by  this  activity.  This 
activity  includes  the  development  of  specifications 
for  procured  subassemblies  and  equipment. 

5.1. 8.1  INPUT  -  STRESS  ANALYSES  AND 
PREDICTIONS 

Output  5.1 .4.6  serves  as  an  input  to  this  activity. 
Quantitative  reliability  requirements  for  all 
production  specifications  are  developed  using  this 
output.  This  output  also  provides  the  data 
necessary  for  the  development  of  production 
specification  derating  and  design  margin 
requirements. 

5.1 .8.2  INPUT  -  MANUFACTURING 
PROCESS  CONTROL  PLAN 

Output  5.1 .6.6  serves  as  an  input  to  this  activity. 
This  output  provides  the  baseline  for  creating 
manufacturing  process  specifications,  identifying 
information,  such  as  SPC  control  variables  and 
control  limits,  and  in-process  inspection 
requirements. 

5.1 .8.3  INPUT  -  TEST  REPORTS 

Output  5.1. 7.3  serves  as  an  input  to  the  production 
specification  activity.  This  output  defines  the 
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baseline  production  verification  and  screening  test 
requirements  lor  subassemblies  and  systems. 

5.1. 8.4  OUTPUT  -  PRODUCT 
PERFORMANCE  SPECIFICATIONS 

This  output  is  the  complete  set  of  DoD  customer 
specifications  and  all  specifications  for  procured 
hardware.  The  specifications  contain  all 
quantitative  and  qualitative  reliability  requirements. 
These  requirements  include  such  parameters  as 
service  life,  Mean  Time  Between  Maintenance 
(scheduled  and  unscheduled)  and  BIT  false  alarm 
rates.  The  specifications  include  such  qualitative 
requirements  as  environmental  descriptions,  design 
margins,  and  parameter  variability  limits.  All  test  and 
performance  verification  requirements  must  be 
defined  in  the  production  specifications. 

5.1 .8.5  OUTPUT  -  MANUFACTURING 
SPECIFICATIONS 

This  output  is  the  specification  of  the  manufacturing 
process,  all  process  steps  are  identified  and 
process  specifications  defined.  These  include 
definition  of  control  variables,  limits  for  those 
variables,  measurement  techniques,  and  the 
definition  of  SPC  requirements.  These 
manufacturing  specifications  should  also  identify  all 
inspection,  test,  reporting,  and  corrective  action 
procedures  applied  to  the  manufacturing  process. 


5.1.9  ACTIVITY  -  DESIGN  REVIEWS  AUDIT  THE  DESIGN 

This  activity  is  substantially  the  same  as  that  defined 
in  paragraphs  3.1.7  and  4.1.7.  Reviews  are  the 
method  for  bringing  discipline  to  the  reliability 
process  and  maintaining  a  focus  on  customer 
requirements.  All  reviews  should  have  clearly 
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defined  entry  and  exist  criteria,  an  explicit  definition 
of  applicable  customer  requirements,  and  a 
description  of  how  these  requirements  are  being 
satisfied.  The  review  process  should  also  have  a 
description  of  responsibilities  for  closure  of  action 
items  and  implementation  of  corrective  action  and 
changes. 


5.1. 9.1  INPUT  -  ACTIVITY  OUTPUTS 

The  outputs  of  Activities  5.1.2,  5.1.4,  5.1.5,  5.1.6, 
5.1.7,  and  5.1.8  define  all  the  reliability  issues  that 
should  be  subject  to  internal  and  customer  review. 
Selected  critical  inputs  (e.g.,  Environment  and  Use 
Description)  to  these  activities  are  also  proper 
issues  for  design  reviews. 

5.1 .9.2  INPUT  -  DESIGN  REVIEW 
PROCEDURES 

The  description  of  this  input  is  the  same  as  provided 
in  paragraphs  3.1 .7.2  and  4.1 .7.2 

5.1. 9.3  OUTPUT  -  DESIGN  REVIEW  REPORT 

The  description  of  this  output  is  the  same  as 
provided  in  paragraph  3. 1.7.3 

5.2  CONTROL  AND  AUDIT 

There  are  three  primary  control  and  audit  metrics 
applicable  during  the  EMD  phase.  Failure  to 
achieve  the  indicated  levels  for  these  metrics  should 
result  in  requirements  for  a  corrective  action  plan. 

5.2.1  DETAIL  QUANTITATIVE  RELIABILITY 
PREDICTIONS 

A  detailed  prediction  of  expected  reliability 
performance  is  one  of  three  control  factors 
applicable  during  the  EMD  phase.  The  current 
practice  of  using  MIL-HDBK-217  for  detailed 
predictions  may  not  be  complete  and  may  require 
supplements  such  as  a  physics  of  failure  approach, 
including  assessments/predictions  of  known 
wearout  and  fatigue  mechanisms,  estimates  of  the 
malperformance  of  the  equipment’s  self-diagnosis 
functions,  and  adjustments  based  on  demonstrated 
results  (customer  use  experience,  part  manufacturer 
test  data). 
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The  credibility  of  the  prediction  is  contingent  on  the 
quality  of  several  key  issues  that  must  be  evaluated 
in  concert  with  a  prediction: 

•  Scope  and  quality  of  “Lessons  Learned" 

•  Design  for  Manufacturing  guidelines  (Design 
simplification) 

•  Environmental  and  Use  data  traceable  to  the 
Mission  Description  (Stresses) 

•  Quality  of  Design  margins/guidelines/derating/ 
application  guidance 

•  Characterization  and  contol  of  Manufacturing 
processes 

•  Materials  characterization  data  (Fatigue/ 
Overstress/Corrosion  resistance) 

•  Parts  control 

Guidance  regarding  techniques  appropriate  to  the 
estimation  of  life  for  some  dominant  fatigue  and 
wearout  mechanisms  are  contained  in  Report  RL-TR- 
91-155,  “Computer  Aided  Assessment  of  Reliability 
Using  Finite  Element  Methods”  and  RL-TR-91-251, 
“Reliability  Assessment  of  Water  Scale  Integration 
Using  Finite  Element  Analysis.”  Sufficient  data 
exists  to  address  additional  issues,  such  as 
connector  durability,  corrosion,  and 
electromigration.  Estimates  of  wearout  and  fatigue 
life  should  be  initially  applied  to  worst  case 
conditions  to  determine  the  need  for  more  extensive 
analyses. 

Use  of  detail  predictions  as  a  control 
parameter  requires  that  corrective  action  be 
implemented  if  the  parameter  falls  below 
established  criteria.  These  criteria  are  as 
follows: 

•  Prediction  values  must  exceed 
customer  requirements  by  a  minimum  of 
twenty  percent  (20%). 

•  Predictions  will  be  iterated  several 
times  during  the  EMD  phase.  Final 
values  shall  reflect  a  twenty  percent 
(20%)  improvement  over  initial  values. 
The  improvement  shall  be  traceable  to 
either  design  simplification  or 
reductions  in  electrical  or  mechanical 
stress  levels. 
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5.2.2  DESIGN  VARIABILITY  CONTROL 

Detailed  quantitative  reliability  predictions  basically 
address  the  issue  of  stress-induced  failures.  The 
robustness  of  the  design  and  its  resistance  to 
parameter  variation  is  another  major  part  of 
eliminating  defects. 

The  basic  design  approach  is  to  keep  the  variation 
under  control  such  that  the  average  result  is 
separated  from  the  specification  limit  by  six  standard 
deviation  units.  This  separation  should  include  the 
effects  of  shifts  and  drifts  in  the  mean.  A  shift  in  the 
mean  of  1 .5  sigma  accounts  for  typical  shifts  and 
drifts.  This  concept  and  a  definition  of  capability 
indices  is  shown  in  Figure  5-11  as  a  measure  of 
control. 
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Figure  5-11.  Concepts  Underlying  Cp  and  Cpk 


The  concepts  Identified  herein  can  be 
applied  to  virtually  any  critical  product 
characteristic.  For  control  purposes  during 
the  EMD  phase,  the  goal  for  each  critical 
parameter  is  six  sigma  (Cp  =  2.0)  with  a 
threshold  for  corrective  action  being  four 
sigma  (Cp  =  1.33). 
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5.2.3  MANUFACTURING  VARIABILITY 
CONTROL 

Manufacturing  is  the  final  element  contributing  to 
defects  and  must  be  “scored”  and  controlled  in  a 
fashion  almost  identical  to  that  described  in 
paragraph  5.2.2. 

The  overall  ability  of  a  manufacturing  process  to 
consistently  produce  a  high-quality  end  item  is 
highly  dependent  on  the  capability  of  the  individual 
steps  that  comprise  that  process.  In  turn,  the 
capability  of  any  given  process  step  is  determined 
by  the  degree  of  capability  related  to,  and  the 
subsequent  control  of,  the  underlying  factors. 

The  control  criteria  for  manufacturing 
control  is  similar  to  that  defined  in 
paragraph  5.2.2.  However,  control  of  the 
manufacturing  process  clearly  extends  into 
both  the  Production  and  Support  phases. 
The  ultimate  goal  for  each  critical 
manufacturing  process  step  is  a  capability 
index,  Cp,  of  2.0.  The  EMO  threshold  for 
corrective  action  is  an  index,  Cp,  of  1.33. 
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Activity  Timeline 


6.1.1 


Quality 

Evaluation 
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Program 
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Figure  6-1.  Production  and  Support  Phases 
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6.0  INTRODUCTION 

This  chapter  defines  the  activities  that  should  take 
place  during  the  Production  and  Operational 
Support  acquisition  phases.  Figure  6-1  shows  the 
essential  activities  of  this  phase  and  the  general  time 
phasing  of  these  activities.  Activity  6.1.1,  quality 
evaluation,  is  a  continuation  of  the  continuous 
improvement  process  started  in  the  Concept- 
Exploration  phase.  It  should  also  be  an  important 
element  of  both  subcontractor  control  and  supplier 
selection.  Activity  6.1.2  is  the  program  planning  for 
both  internal  engineering  and  manufacturing 
operations  and  customer  support  for  fielded  systems. 

Activities  6.1.3  and  6.1.4  are  the  internally  and 
externally  directed  actions  required  for  defect 
detection,  elimination,  and  prevention. 

Continuing  design  and  manufacturing  reviews, 
activity  6.1.5,  are  essential  to  maintain  a  focus  on 
both  continuous  improvement  and  continuing 
satisfaction  of  customer  requirements. 

Specific  descriptions  of  the  phase  activities  are  MAJOR  CHANGES 

contained  in  paragraphs  6.1.1  through  6.1.5.  These 

paragraphs  discuss  the  routine  reliability 

improvement  activities  of  the  production  and 

operational  support  phases.  Major  changes, 

upgrades,  or  improvements  are  subject  to  control  via 

the  appropriate  activities  from  the  previously 

described  phases.  The  activities  are  selected 

based  on  the  development  status  of  the  change. 

6.1  SUMMARY  OF  ACTIVITIES 

The  full  scale  production  phase  implements  the 
contractor’s  production  plan,  as  modified,  by  what 
was  learned  in  the  preproduction  phase.  Using 
Statistical  Process  Control  (SPC)  to  measure  the 
process  and  product  variability  at  strategic  points  in 
the  production  flow  will  ensure  that  reliability  is  not 
affected  by  workmanship,  tooling  tolerances, 
equipment  calibration,  and/or  various  manufacturing 
processes.  Once  the  processes  are  under  control, 
the  next  most  important  task  is  to  ensure  that  all 
anomalies  are  properly  corrected.  This  is 
accomplished  through  the  Failure  Reporting 
Analysis  and  Corrective  Action  System  (FRACAS) 
process.  The  data  collected  from  this  system  is  used 
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to  establish  failure  history,  cause,  and  corrective 
action.  It  provides  the  detail  necessary  to  establish 
trends  and  provide  closed  loop  feedback  to  the 
designer  on  product  and  process  problems  that 
require  modification. 

Verification  test  data  is  collected  from  production 
Environmental  Stress  Screening  (ESS)  and 
Production  Reliability  Acceptance  Tests  (PRAT)  to 
ensure  that  parts  and  materials  and  manufacturing 
processes  provide  what  is  necessary  to  produce  a 
product  that  is  consistently  reliable. 

Once  the  product  is  fielded,  the  focus  is  on  failure 
reporting,  analysis,  and  corrective  action.  Failure 
reporting  analysis  and  corrective  action  data  is 
carried  over  from  production  and  combined  with 
actual  field  performance  data  to  determine  if  there 
are  unique  conditions  that  must  be  addressed  that 
did  not  appear  in-house. 

It  is  critical  that  information  pertaining  to  performance 
criteria,  such  as  range,  accuracy,  clarity,  speed, 
reliability,  etc.,  is  accurately  reported. 

Customer/User  satisfaction  must  be  primary,  and  if 
an  anomaly  does  occur,  rapid  response  is  important. 
Information  provided  can  be  used  to  isolate  design 
deficiencies  or  unforeseen  process  problems  that 
don't  show  up  until  the  product  is  in  the  field.  This 
information  is  then  fed  back  to  the  contractor  to 
determine  the  root  cause  and  implement  corrective 
action.  Data  collected  and  stored  over  the 
operational  life  of  the  program  can  reveal  long  term 
conditions.  This  information  may  reveal  handling  or 
usage  conditions  that  can  be  compensated  for  in 
existing  and  future  designs. 

The  following  program  attributes  must  be  in  place  for 
the  Production  and  Operational  Support  phases: 

•  An  effective  system  that  translates  all  the 
customer’s  expectations  and  requirements  into 
Production  and  Operational  Support 
requirements. 

•  An  effective  system  for  identifying  key  product 
and  process  characteristics  and  their  impact 
on  reliability/durability. 
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•  An  effective  system  of  process  controls  that 
addresses  process  yield  and  variability  and  an 
inhouse  feedback  system  that  informs 
engineering  of  potential  design  deficiencies. 

•  An  effective  plan  that  verifies  through  testing 
the  life  and  reliability  of  the  product  that  gives 
credibility  to  earlier  predictions. 

•  An  effective  system  of  reporting,  retrieving, 
analyzing,  and  correcting  field  problems  that 
might  have  a  relationship  to  product  design  or 
manufacturing  processes. 

With  these  attributes  in  place,  and  with  the 
application  of  TQM  tools,  such  as  Cycle  Time 
Management  and  SPC,  the  production  phase  should 
have  good  product  yield  with  exceptional  reliability 
and  durability.  Operational  Support  costs  will  be  low 
when  the  product  is  fielded,  and  customer/user 
satisfaction  will  be  high. 


6.1.1  ACTIVITY  -  QUALITY  EVALUATION 

This  activity  shifts  during  this  phase  from  a  primary 
use  as  source  selection  criteria  to  an  evaluation 
directed  at  monitoring  and  measuring  continuous 
improvement.  However,  it  can  and  should  still  be 
used  for  source  selection  for  competitive 
reprocurement  and  second  sourcing.  The  quality 
evaluation  should  also  be  embedded  in  a  supplier 
certification  program  directed  at  identifying  a 
selection  of  preferred  suppliers.  The  quality 
evaluation  should  also  be  used  as  a  measurement 
tool  for  continuous  improvement,  with  results 
requested  periodically  and  reviewed  as  part  of 
supplier  monitoring  and  control. 
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6.1. 1.1  INPUT  -  EVALUATION  CRITERIA 

The  evaluation  criteria  defined  in  Volume  II  of  this 
handbook  is  the  recommended  benchmark  for 
continuing  quality  evaluation.  These  criteria  should 
be  included  in  production  contracts  with  instructions 
for  periodic  evaluations. 

6.1 .1.2  OUTPUT  -  QUALITY  EVALUATION  ASSESSMENT 

REPORT  REPORTS 

The  supplier  provides  a  quality  evaluation  report  in 
accordance  with  customer  reporting  requirements. 

The  Malcolm  Baldrige  report  format  is  a 
recommended  source  for  reporting  requirements. 

These  reports  should  be  prepared  and  evaluated  on 
an  annual  or  biannual  basis  with  a  focus  on  problem 
areas  and  corrective  actions. 


6.1.2  ACTIVITY  -  PROGRAM  PLANNING  FOCUS  OF 

This  activity  is  reduced  in  comparison  to  the  ACTIVITIES 

planning  required  for  prior  acquisition  phases.  The 

planning  for  the  production  and  operational  support 

phases  should  emphasize  the  continuing  reduction 

in  variability  of  manufacturing  processes,  failure 

reporting,  analyses  and  corrective  action  systems, 

supplier  control,  and  the  development  of  customer 

use  feedback  systems.  The  plans  must  clearly 

identify  the  relationships  between  reliability 

engineering,  manufacturing,  quality  assurance, 

design  engineering,  and  logistic  support 
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organizations. 

6.1. 2.1  INPUT  -  FUNDING  PROFILES 

The  description  of  this  input  is  the  same  as 
contained  in  paragraphs  3.1. 3.1,  4.1 .3.1,  and 
5.1. 3.1. 


6.1. 2.2  INPUT  -  PRODUCT  PERFORMANCE 
SPECIFICATIONS 

The  output  described  in  paragraph  5.1 .8.4  serves  as 
an  input  to  the  planning  activity.  These 
specifications  completely  describe  all  performance 
and  verification  requirements. 

6.1. 2.3  INPUT  -  MANUFACTURING 
REQUIREMENTS 

The  output  described  in  paragraph  5. 1.6.5  serves  as 
an  input  to  the  planning  activity.  This  input  includes 
all  manufacturing  process  specifications,  the  status 
of  capability  indices  (Cp)  for  the  design  and 
manufacturing  processes,  and  SPC  criteria. 

6.1. 2.4  INPUT  -  CONTRACT 
REQUIREMENTS 

The  description  of  this  input  is  the  same  as  provided 
in  paragraph  3. 1.3.2. 

6.1. 2.5  OUTPUT  -  TAILORED  PROGRAM 
PLAN 

The  program  plan  should  use  the  inputs  and  outputs 
of  Activities  6.1.1,  6.1.3,  6.1.4,  and  6.1.5,  as  the 
baseline  for  defining  the  specific  tasks  for  the 
production  and  operational  support  phases.  The 
plan  should  concentrate  on  the  topics  described  in 
paragraph  6.1.2. 

6.1. 2.6  OUTPUT  -  FAILURE  REPORTING  ELIMINATE  DEFECTS 

ANALYSIS  AND  CORRECTIVE  ACTION 

SYSTEM  (FRACAS) 

The  supplier  must  have  a  comprehensive  FRACAS 
that  addresses  all  defects  discovered  in  both  the 
manufacturing  and  test  processes.  The  system 
should  clearly  identify  the  requirements  and 
responsibility  for  reporting,  analyzing,  correcting, 
and  tracking  the  status  of  all  defects.  There  are  two 
keys  to  the  effectiveness  of  FRACAS  systems:  1)  it 
should  be  a  single  system  that  records  and  corrects 
defects  from  all  sources  from  receipt  of  parts  and 
materials  through  finished  product,  and  2)  there  must 
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be  evidence  of  continuous  management  oversight 
and  insistence  on  the  root  cause  analysis  and 
correction  of  all  defects. 

6.1. 2.7  OUTPUT  •  BENCHMARKING  AND 
TRAINING  PLAN 

Benchmarking  plans  should  be  built  around  two 
issues:  1 )  weaknesses  uncovered  during  the  quality 
evaluation  activity  of  paragraph  6.1.1,  and  2)  the 
issues  of  continuing  variability  reduction,  failure 
elimination,  and  supplier  control.  The  plans  should 
identify  the  process  and  company  being 
benchmarked,  the  data  collection  method, 
responsibility  for  the  effort,  and  schedules  for 
completing  the  benchmarking.  Suggested  topics  for 
benchmarking  include: 

•  Failure  Reporting,  Analyses,  and  Corrective 
Action  Systems 

•  Customer  Feedback  Systems 

•  Manufacturing  Cycle  Time 

•  Supplier  Control 

•  Statistical  Process  Control 

•  Variability  Reduction 

The  training  plan  should  address  the  capability  of 
program  personnel  to  accomplish  the  phase 
activities,  inputs  and  outputs,  and  the  training 
resources  employed  to  correct  deficiencies.  The 
training  plan  should  also  address  the  topic  of  the 
dissemination  of  reliability  requirements  throughout 
all  program  areas. 

6.1. 2.8  OUTPUT  -  CUSTOMER  FEEDBACK 
PLAN 

Customer  feedback  is  a  critical  element  during  the 
Production  and  Operational  Support  phases.  The 
supplier  should  develop  a  plan  for  obtaining 
information  from  their  customer  and  the  end  use  of 
the  equipment.  The  plan  should  identify  all  use 
environments,  the  type  of  data  generated,  and 
procedures  for  the  disposition  and  repair  of  failed 
items.  Having  identified  these  issues,  the  plan 
should  describe  the  methods  employed  to  collect  all 
relevant  information,  the  responsibility  for  acting  on 
the  information,  and  the  status/tracking  techniques 
to  be  applied  to  ensure  problem  resolution.  The  plan 
should  address  both  the  customer’s  data  and  the 
mechanics  for  transmitting  this  information  to 
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suppliers  and  subcontractors  tor  their  action  and 
problem  closeout. 


6.1.3  ACTIVITY  -  PRODUCTION  IN-PROCESS  TESTING 

RELIABILITY  CONTROL 

This  activity  is  a  central  activity  of  the  Production 
and  Operational  Support  phases.  Data  from  internal 
manufacturing  processes,  supplier  processing  and 
test,  and  all  pre-delivery  testing  is  evaluated.  Based 
on  these  analyses,  changes  to  manufacturing 
processes,  parts,  and  materials  are  implemented. 

The  key  to  the  success  of  this  activity  is  the 
comprehensiveness  of  the  data  sources. 

6.1 .3.1  INPUT  -  MANUFACTURING  MANUFACTURING 

PROCESS  DESCRIPTION  RESULTS 

This  is  a  comprehensive  definition  of  the 
manufacturing  process.  The  data  includes  a 
description  of  every  manufacturing  process  step,  the 
capability  index  associated  with  that  step,  the 
identification  of  inspection  and  test  steps,  and  the 
data  reflecting  the  number  of  defects  and  parameter 
variability  detected  at  each  of  the  test  and 
inspection  steps.  This  latter  data  should  be 
represented  in  the  FRACAS  data  base.  Within  the 
description  of  the  manufacturing  process  steps, 
those  affecting  critical  product  characteristics 
should  be  identified  and  highlighted. 
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6.1. 3.2  INPUT  -  SUPPLIER  CONTROL 
RESULTS 

Oata  from  suppliers  of  critical  parts  and  materials 
should  maintain  the  data  indicated  in  paragraph 
6. 1.3.1  and  provide  variability  data  for  critical  parts 
and  materials  parameters.  This  input  also  includes 
parts  and  materials  defect/failure  data  and  status 
reports  for  problem  closeout. 

6.1 .3.3  INPUT  -  PRE-DELIVERY  TEST 
RESULTS 

Pre-delivery  testing  includes  all  post-manufacturing 
assembly  testing,  including  Environmental  Stress 
Screening  at  all  levels  of  assembly,  acceptance 
testing,  and  any  operation  or  testing  prior  to  use  by 
the  ultimate  customer.  These  data  become  part  of 
the  defect  data  base  subject  to  reporting,  analyses, 
corrective  action,  and  problem  closure  status 
tracking.  During  this  testing,  there  should  be  no 
acceptable  levels  of  defect  nor  any  defects  which 
are  not  subject  to  corrective  action  requirements. 

All  testing  should  be  identified  in  either  performance 
or  manufacturing  specifications. 

6.1 .3.4  OUTPUT  •  MANUFACTURING 
PROCESS  CHANGES 

Test  results  from  all  phases  of  testing  should  be 
combined  and  used  to  validate  the  achievement  of 
six  sigma  levels  in  both  design  and  manufacturing 
processes.  The  capability  index  of  the 
manufacturing  processes,  starting  with  critical 
processes,  should  be  demonstrated  to  be  at  least  2.0 
(six  sigma).  Planned  corrective  actions,  such  as 
variability  reduction  testing,  SPC,  and  process 
simplification,  should  be  defined  for  processes  not 
achieving  necessary  levels  of  reliability. 

6.1. 3.5  OUTPUT  -  PARTS  AND  MATERIALS 
CHANGES 

Test  data  from  all  phases  of  testing,  including  tests 
conducted  by  parts  and  materials  suppliers,  should 
be  combined  to  demonstrate  the  achievement  of  six 
sigma  quality  levels  by  parts  and  ma'  Ms  supplier 
manufacturing  processes.  In  addition,  test  data 
should  be  accumulated  on  critical  parameters  to 
validate  that  variability  relative  to  required 
tolerances  are  equal  to  six  sigma.  Plans  for 
corrective  action  should  be  developed  for  parts  and 
materials  not  complying  with  these  requirements. 
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Within  the  constraints  of  configuration  management, 
the  design  should  be  continuously  reviewed  for 
simplification  and  the  incorporation  of  improved 
technology  parts  aid  materials. 


6.1.4  ACTIVITY  -  CUSTOMER  SUPPORT 

Satisfaction  of  the  ultimate  using  customer  is  the  goal 
of  this  activity.  Provisions  should  be  in  place  that 
ensure  that  all  levels  of  suppliers  are  made  aware  of 
the  performance  of  their  products  in  the  hands  of  the 
end  user.  Satisfaction  should  be  measured  both  by 
conformance  to  performance  specification 
requirements  and  by  direct  customer  contact. 

6.1 .4.1  INPUT  -  LOGISTIC  SUPPORT  PLAN 

The  support  environment  and  the  stresses  resulting 
from  that  environment  have  been  identified  as 
design  requirements  from  the  earliest  acquisition 
phases.  This  input  represents  data  describing 
troubleshooting  and  repair  practices  and 
environments  at  all  levels  of  maintenance,  support 
and  test  equipment  at  all  levels  of  maintenance, 
storage  handling  and  transportation  conditions,  and 
maintenance  training.  The  data  also  includes  details 
regarding  procedures  for  warranties  and  return  of 
failed  assets  to  suppliers  for  repair. 

6.1. 4.2  INPUT  -  CUSTOMER  DATA 
FEEDBACK 

All  sources  of  performance  data  should  be  identified 
and  employed  to  measure  customer  satisfaction. 

This  data  includes  standard  DoD  maintenance  data, 
depot  repair  reports,  direct  customer  contact,  and 
reports  from  contractor  field  service  personnel.  This 
data  should  be  collected  and  evaluated  in  a 
centralized  and  systematic  fashion. 


PRODUCT 

PERFORMANCE 


FIELD  DATA 
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6.1. 4.3  OUTPUT  -  DESIGN  CHANGES 

Data  analyses  should  be  a  continuous  supplier 
process  to  measure  quantitative  reliability 
performance  levels,  identify  problem  areas,  and 
validate  assumptions  regarding  customer 
environments  and  use.  The  supplier  should  have  a 
process  in  place  that  ensures  the  widest  possible 
distribution  of  these  analytic  results  along  with 
criteria  for  initiating  design  changes  to  correct 
problems. 


6.1 .4.4  OUTPUT  -  MANUFACTURING 
PROCESS  CHANGES 

The  description  of  this  output  is  substantially  the 
same  as  output  6.1 .4.3  except  that  the  output  product 
is  manufacturing  process  changes. 


6.1.5  ACTIVITY  -  DESIGN  AND  REVIEW  THE  RESULTS 

MANUFACTURING  REVIEWS 

The  supplier  should  have,  in  place,  a  process  for 
ensuring  the  continuous  review  of  the  results  of 
activities  6.1.1,  6.1.3,  and  6.1.4.  The  review 
process  includes  both  internal  and  customer 
reviews.  All  reviews  should  have  clearly  defined 
entry  and  exit  criteria,  an  explicit  definition  of 
applicable  customer  requirements,  and  a 
demonstration  that  these  requirements  are  being  met. 

The  review  process  is  an  integral  part  of  outputs 
6.1. 3.4,  6.1 .3.5,  6.1. 4.3,  and  6.1. 4.4. 
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6.1 .5.1  INPUT  -  MANUFACTURING 
PROCESS  CAPABILITY  INDICES 

This  input  defines  the  variability  of  key 
manufacturing  processes  in  terms  of  process 
capability  indices  (Cp  or  Cpi<)  or  defects  per  million 
opportunities. 

6.1. 5.2  INPUT  -  PRE-DELIVERY  TEST 
RESULTS 

Input  6. 1.3.3  also  serves  as  an  input  to  the  review 
activity. 

6.1 .5.3  INPUT  -  CUSTOMER  DATA 
FEEDBACK 

Input  6.1. 4.2  also  serves  as  an  input  to  the  review 
activity. 


6.1. 5.4  OUTPUT  -  PRODUCT 
IMPROVEMENT  PROPOSALS 

One  of  the  principal  outputs  of  the  review  process 
are  recommendations  for  product  improvements. 
These  are  product  changes  that  require  customer 
approval  or  those  requiring  any  contractual  or 
configuration  changes. 

6.1 .5.5  OUTPUT  -  SUPPORT  SYSTEM 
CHANGE  PROPOSALS 

These  outputs  represent  changes  in  the  customer’s 
support  activities  and  require  customer  approval. 

6.2  CONTROL  AND  AUDIT 

The  two  control  metrics  for  this  phase  are 
the  six  sigma  manufacturing  capability 
metrics  described  in  paragraph  5.2,  and 
product  performance  specification  reliability 
values  which  should  be  achieved  in  the 
user’s  environment.  This  is  supplemented 
by  demonstration  of  equipment/  system  level 
performance  at,  or  above,  customer 
requirements  during  pre-delivery  testing. 

The  recommended  methodology  for  tracking 
the  achievement  of  customer  requirements, 
based  on  pre-delivery  testing,  is  described 
in  paragraph  2.8,  pages  83  through  88,  of 
Report  RL-TR-91-300,  Volume  1  (of  2), 
“Evaluation  of  Quantitative  Environmental 
Stress  Screening  (ESS)  Methods.”  The  only 
stipulation  regarding  this  procedure  is  that 


METRICS 
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all  defects  and  equipment  malperformance, 
including  design  errors  and  malperformance 
of  self-diagnostics,  be  included  as  part  of 
the  data  base. 
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Figure  7*1.  Pre-Concept  Exploration  Phase  Quality  Function  Deployment 
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7.0  INTRODUCTION 

This  example  demonstrates  how  the  reliability 
process  can  be  used  to  enhance  the  reliability  of 
products.  It  by  no  means  covers  all  the  areas  of  the 
reliability  process,  but  instead  covers  some 
important  areas  in  a  concrete  manner  as  a  general 
demonstration  of  the  use  of  the  process.  The  overall 
product  in  this  example  is  a  radar  system,  with  most 
of  the  example  concentrating  on  an  identified  critical 
item,  a  linear  hybrid.  Even  though  this  example 
centers  on  a  linear  hybrid,  the  process  can  be  used 
on  all  products  and  at  all  levels  of  indenture. 
Samples  of  the  use  of  Quality  Function  Deployment 
(QFD)  are  integrated  into  the  example.  These  show 
how  QFD  is  used  to  interpret  customer  needs  and 
flow  the  resulting  technical  requirements  to  the 
lowest  levels  of  design  and  manufacturing. 

7.1  PRECONCEPT  EXPLORATION  PHASE 
ACTIVITIES 

Before  the  formal  request  for  proposal  (RFP)  was 
released,  Ajax  Company  met  with  the  customer  to 
discuss  their  needs  and  expectations  for  the  future 
Radar  system  and  to  work  with  the  customer  in 
defining  requirements  and  explaining  contractor 
capability.  The  translation  of  customer  needs  into 
requirements  was  performed  using  QFD.  Figure  7-1 
shows  an  example  of  a  partial  QFD.  The  customer's 
operational  needs  are  translated  into  system  level 
technical  objectives.  These  objectives  are  related 
to  customer  needs  and  priorities  and  "scored"  to 
define  the  relative  importance  of  the  objectives.  At 
this  early  stage  in  the  acquisition  cycle,  the  QFD 
technical  objectives  (labeled  in  the  Figure  as 
Design  Requirements)  can  be  used  as  criteria  for 
selecting  emerging  technologies  that  may  satisfy 
these  objectives.  The  Ajax  Company  had  used  this 
approach  on  other  contracts  and  found  it  most 
beneficial.  In  the  past,  the  customer  passed  down 
requirements  that  were  beyond  Ajax’s  capability  or 
did  not  directly  translate  into  meeting  total  customer 
requirements.  By  discussing  the  expectations  and 
needs  of  the  customer  before  the  RFP  was  released, 
Ajax  was  able  to  inform  the  customer  of  Ajax’s 
technical  capability  and  options,  and  work  with  them 
to  develop  requirements  that  would  directly  translate 
into  satisfying  their  needs  and  expectations. 
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Figure  7-2.  Concept  Exploration  Phase  Quality  Function  Deployment 
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Upon  completing  the  requirements  development, 

Ajax  concluded  that  they  had  the  capability  to  meet 
the  needs  and  expectations  of  the  customer.  Ajax 
was  currently  building  three  radar  systems:  an  APG 
50,  51 M,  and  52M  Radar.  The  51 M  and  52M  radar 
both  utilized  slightly  different  design  concepts 
compared  to  the  APG  50.  However,  the  51 M  and 
52M  radars  utilized  a  Radar  Receiver  Processor 
(RRP)  module  that  toe  designer  felt  was  optimal  for 
the  upgrade  of  the  APG  50.  Ajax’s  plan  was  to 
redesign  the  receiver  module  in  the  APG  50  Radar 
using  the  receiver  (RRP)  module  from  the  51 M  or 
52M  Radar.  However,  a  Preconcept  Risk  Analysis 
found  that  toe  receiver  module  has  been  the  problem 
module  in  the  factory  and  in  the  field.  The  problem 
was  determined  to  be  the  Receiver  (RCV)  hybrid. 

The  RCV  hybrid  is  the  heart  and  soul  of  the  RRP 
module. 

An  alternative  approach  considered  by  Ajax 
Company  was  to  redesign  the  RRP  module  using  the 
Super  Receiver  (SRCV I)  hybrid.  The  SRCV I  hybrid 
was  part  of  a  new  program  funded  to  design  and 
build  cost  effective  state-of-the-art  receiver  hybrids 
using  Microwave  Monolithic  Integrated  Circuits 
(MMIC)  GaAs  chips  and  tape  automated  bonding.  A 
detailed  trade-off  study  was  planned  for  the  Concept 
Phase  to  investigate  the  advantages  and 
disadvantages  of  the  redesigned  RCV  hybrid  versus 
the  new  technology. 

7.2  CONCEPT  EXPLORATION  PHASE 
ACTIVITIES 

Upon  receiving  the  RFP,  Ajax  Company  began  the 
formal  development  of  the  proposal.  The  customer 
requirements  stated  in  the  contract  were  very  similar 
to  the  requirements  that  were  developed  during  the 
preconcept  phase.  The  analysis  and  deployment  of 
requirements  from  the  preconcept  phase  were 
updated  using  Quality  Functional  Deployment  QFD 
to  establish  cost,  performance,  reliability, 
producibility,  and  support  requirements,  along  with 
identifying  customer  priorities  and  requirement 
interaction.  As  shown  in  Figure  7-2,  the  technical 
objectives  represent  system  level  objectives,  but 
contain  additional  and  more  specific  attributes  than 
identified  during  Pre-Concept  Exploration  phase. 
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Their  winning  proposal  included  a  plan  that  clearly 
demonstrated  how  the  customer’s  needs  and 
expectations  would  be  satisfied.  Ajax  attributed 
much  of  the  success  to  the  Preconcept  QFD,  which 
established  what  the  customer  was  looking  for  and 
translated  these  needs  into  requirements  that  both 
parties  felt  were  optima). 

The  Concept  Phase  Reliability  Implementation  Plan 
included  a  general  outline  of  the  activities  planned 
through  Production.  Though  many  of  the  details  of 
the  plan  were  not  known  at  toe  time,  the  general 
concepts  were  well  defined.  The  general  areas 
covered  were  roles  and  responsibilities,  design 
guides  development,  critical  item  identification, 
variability  control,  and  verification  test.  An 
engineering  and  manufacturing  team  collectively 
established  the  specific  roles  and  responsibilities  tor 
contract  activities. 

In  order  to  establish  a  system  concept  and  a  firm 
direction  for  choice  of  technology,  toe  designer 
requested  additional  historical  data  to  confirm  the 
poor  reliability  history  of  the  RSV  hybrid.  The 
investigation  determined  that  many  of  the  critical 
parts  and  processes  of  the  RSV  hybrid  had  been 
identified  and  corrected,  but  in  spite  of  this,  toe 
hybrid  continued  to  demonstrate  poor  reliability 
performance.  Concurrently,  he  contacted  toe  new 
technology  SRSV I  hybrid  program  to  determine 
what  actions  had  been  taken  to  determine  critical 
parts  and  processes  for  the  SRCV I  hybrid.  There 
had  been  limited  analyses  or  tests  performed  to 
determine  toe  critical  materials  or  processes  of  these 
new  technologies.  The  cost  of  performing  these 
analyses  and  tests  were  available  from  toe  SRSV  I 
hybrid  program  manager.  The  design  engineer 
planned  to  use  these  for  his  Trade-Off  Analysis. 

The  management  of  Ajax  had  recently  adopted  a 
new  philosophy  of  commonality  of  designs  across 
programs.  This  was  another  consideration  for  the 
Trade-Off  Analysis.  The  RSV  hybrid  was  already 
used  on  the  M50  and  M51  program  and  many  of  the 
critical  processes  of  the  RSV  hybrid  were  already 
known.  Little  was  known  about  the  reliability  of  the 
SRSV  I  hybrid  and  the  development  costs  were  high. 
When  these  factors  were  used  in  the  Trade-Off 
Analysis,  it  was  decided  that  redesigning  the  RRP 
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module  using  the  RSV  hybrid  was  the  optimal 
decision.  It  was  a  dear  choice  based  on  the  data 
available. 

Once  tiie  configuration  and  technology  choice  was 
decided,  a  risk  reduction  plan  was  developed  to 
improve  the  reliability  of  the  identified  critical  item, 
the  RSV  hybrid. 

7.3  DEMONSTRATION  AND  VALIDATION 
PHASE  ACTIVITIES 

At  the  completion  of  the  concept  phase,  Ajax  had 
identified  a  critical  item,  the  RCV  hybrid  and  a  risk 
reduction  plan.  This  example  will  now  concentrate 
on  applying  the  reliability  process  to  the  RCV 
hybrid.  Remember,  there  may  be  many  critical  items 
in  a  design.  This  example  demonstrates  just  one  of 
the  many  ways  of  applying  the  reliability  design 
process. 

The  analysis  and  deployment  of  requirements 
(Quality  Functional  Deployment)  was  used  to 
flowdown  cost,  performance,  reliability, 
producibility,  and  support  requirements-first  to  the 
Receiver  module,  and  then  to  the  critical  component 
(Figures  7-3  and  7-4).  At  the  component  level,  the 
QFD  resulted  in  very  specific  technical 
requirements  and  actions  focused  on  defining  and 
correcting  the  technical  problems  of  the  RCV  hybrid. 
Important  requirements  for  this  example  were  stress 
cycle  life  process  yields  and  defect  rate. 

During  the  Concept  Exploration  phase,  the  RCV 
hybrid  was  chosen  as  the  preferred  design  option, 
despite  the  fact  that  early  critical  parts  review 
indicated  a  poor  reliability  history.  The  designer 
took  this  information  into  account  in  his  DEM/VAL 
Reliability  Plan.  The  reliability  engineer, 
manufacturing  engineer,  component  specialist,  and 
the  design  engineer  jointly  developed  the  following 
risk  reduction  plan  details: 

1 .  Perform  a  critical  parts  and  materials  review  for 
the  hybrid.  Look  at  the  historical  data  in  detail. 
Determine  use  conditions  when  the  device 
failed  and  what  was  the  physical  failure  mode. 
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2.  Determine  if  failure  analyses  had  been 
performed  to  determine  cause  of  failure  or 
develop  accelerated  teste  and  failure  analyses 
to  determine  the  dominant  stress  factors  and 
the  root  cause  of  the  failure. 

Upon  performing  the  critical  parts  and  materials  REDUCE  RISK 

review,  it  was  confirmed  that  the  hybrid  had 

demonstrated  poor  reliability  performance,  both  in- 

house  and  in  the  field,  on  two  different  Ajax 

programs.  Reliability  data  showed  that  the  hybrid 

failed  to  meet  toe  minimum  standards  for  critical 

process  yields  and  the  field  failure  rate  was 

excessive.  Review  of  the  data  indicated  that  toe 

failure  mode  was  possibly  related  to  temperature 

cycling.  Because  limited  failure  analysis  had  been 

performed  on  failed  hybrids,  accelerated  teste  and 

failure  analyses  were  conducted  to  determine  toe 

dominant  stress  factors  and  the  root  cause  of  failure. 

The  results  concluded  that  the  cause  of  failure  was 
related  to  the  mismatch  of  the  thermal  coefficient  of 
expansion  (TCE)  of  the  semiconductor  die  attach 
material  and  the  carrier  it  was  mounted  on. 

The  mechanism  of  this  failure  was  determined  to  be 
fatigue  of  the  epoxy  die  attach  material,  which  is 
temperature  cycling  dependent.  The  thickness  of 
the  epoxy  die  attach  material  was  also  determined  to 
be  important  for  thermal  dissipation  needs. 

The  designer  went  back  to  the  failure  modes  and 
effects  analysis  (FMECA)  developed  in  the  earlier 
part  of  the  Demonstration  and  Validation  Phase  to 
update  the  analysis  to  include  the  new  failure 
information.  The  break  in  the  thermal  path  through 
the  epoxy  caused  the  semiconductor  die  to  overheat 
and  fail.  The  FMECA  showed  that  failure  of  this  chip 
would  result  in  a  single  point  failure  to  the  hybrid  and 
to  the  overall  system. 

The  test,  analyses,  and  conclusions  were  sufficient 
to  show  that  reliability  problems  could  be  resolved 
during  the  Engineering  and  Manufacturing 
Development  phase. 

7.4  ENGINEERING  AND  MANUFACTURING 
DEVELOPMENT  (EMD)  PHASE  ACTIVITIES 

In  this  example,  the  transition  to  EMD  was 
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approximately  continuous.  The  customer  had 
reviewed  the  Reliability  Plan  at  the  conclusion  of  the 
Demonstration  and  Validation  phase  and  requested 
an  update  to  the  Reliability  Plan  to  explain  what 
actions  would  be  taken  to  address  the  hybrid 
reliability  issues. 

The  QFD  had  been  continually  updated  from  the 
start  of  the  program,  so  toe  flowdown  of  customer 
expectations  and  requirements  went  very  smoothly. 

Figure  7-5  showc  the  final  flowdown  of  requirements 
for  the  RCV  hybrid.  The  updates  had  been  rigorous, 
and  comments  at  the  design  review  resulted  in  very 
few  requirement  changes.  EMD  Master  Schedules 
confirmed  the  value  of  performing  a  preliminary 
critical  parts  and  material  review  during  DEM/VAL. 

Since  the  time  envelope  available  for  determining 
critical  items  and  developing  a  course  of  action 
during  EMD  was  very  short,  it  was  most  beneficial  to 
determine  critical  issues  as  early  as  possible  in  the 
design  process  to  allow  time  for  refinement  and  final 
optimization. 

Upon  reviewing  the  requirement  changes  from 
DEM/VAL  to  EMD  and  the  DEM/VAL  design  and 
analysis  data,  it  was  determined  that  additional 
testing  and  analysis  was  required.  The  EMD 
planning  effort  considered  all  DEM/VAL  information 
and  included  additional  planning,  testing,  and 
analysis  that  was  required  to  ensure  the  design 
would  meet  all  program  requirements.  The  Reliability 
Plan  was  developed  in  a  concurrent  engineering 
environment  using  QFD  to  aid  in  the  prioritization  of 
activities.  The  design  engineer,  component 
specialists,  failure  analysts,  manufacturing 
engineers,  and  reliability  engineers  jointly 
developed  the  following  hybrid  improvement  plan: 

1.  Consult  with  material  specialists  and  DEFECT  PREVENTION 

component  engineers  in  choosing  potential 

epoxy  suppliers.  Choose  the  epoxies  to  be 
evaluated  based  on  the  appropriate  thermal 
coefficient  of  expansion  (TCE),  other  thermal 
characteristics,  historical  life  and  producibility 
data,  and  cost. 

2.  Use  thermal  modeling  techniques  to  establish  a 
maximum  thickness  for  each  epoxy  type. 
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Figure  7-5.  Engineering  and  Manufacturing  Development  Phase  Quality  Function  Deployment 
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3.  Using  engineering  judgement,  historical  data, 
and  vendor  experience,  determine  a  minimum 
epoxy  die  attach  thickness  as  a  target  value  for 
each  epoxy  type.  Choose  three  thicknesses 
within  toe  feasible  range. 

4.  Utilize  Design  of  Experiments  (DOE)  for 
performing  toe  following  manufacturing 
process  development  tests:  a)  determine  the 
relationship  between  the  amount  of  epoxy  and 
the  epoxy  die  attach  thickness  after  curing  for 
each  epoxy  type,  including  the  corresponding 
values  for  the  application  needle  height  (mils), 
robot  speed  (cm/s),  and  needle  diameter  (mils), 
as  well  as  die  placement  pressure;  b) 
determine  process  control  variables  for 
applying  the  desired  quantity  of  each  epoxy 
type. 

5.  Choose  five  different  temperature  ranges  that 
are  within  the  application  requirement  and 
perform  temperature  cycling  to  failure. 

6.  Use  DOE  to  evaluate  the  variability  of  cycles 
to  failures  for  each  epoxy  type,  based  on 
thickness,  temperature  range,  and  epoxy 
supplier. 

7.  Determine  Key  Product  Characteristics  and 
Key  Control  Characteristics. 

8.  Develop  a  Statistical  Process  Control  Plan. 

Based  on  previous  experience,  the  material 
specialist  and  component  engineer  were  able  to 
choose  three  potential  epoxies,  all  supplied  by 
different  suppliers.  The  three  epoxies  were  chosen 
for  further  studies  based  on  TCE  and  thermal 
characteristics,  historical  life  and  producibility  data, 
and  cost. 

Because  of  early  planning,  many  of  the  activities 
stated  in  the  Reliability  Plan  were  performed 
simultaneously,  instead  of  toe  chronological  order 
stated  above.  This  procedure  optimized  the  time 
available  for  reliability  product  development.  While 
thermal  modeling  was  being  performed  to  determine 
toe  maximum  thickness  to  satisfy  thermal  dissipation 
targets,  samples  were  being  developed  for  testing, 
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process  development,  and  produdbility  studies. 

Upon  completing  steps  1  through  7,  results  were 
weighted  by  order  of  criticality  and  used  to  perform  a 
tradeoff  of  reliability,  manufacturability,  and  cost 
The  trade-off  activity  resulted  in  a  choice  of  Epoxy  1 . 
Epoxy  1  was  subjected  to  a  reliability  verification 
test.  All  Epoxy  1  samples  passed  all  the  verification 
tests.  The  desired  epoxy  die  attach  thickness  was 
determined  to  be  (1  mil  <  X  <  1 .5  mil).  This  was 
identified  as  a  Key  Product  Characteristic  (KPC). 

The  required  quantity  of  Epoxy  1  to  satisfy  the 
thickness  requirement  was  determined  to  be  (2  gms 
<  X  <  2.3  gms).  This  was  identified  as  a  Key  Control 
Characteristic  (KCC),  along  with  a  die  placement 
pressure  of  (5  gms  <  X  <  6  gms). 

The  epoxy  die  attach  thickness  is  critical.  It  was 
determined  through  analysis  that  if  it  varies  outside 
of  the  determined  thickness  requirement,  a  failure 
will  occur.  The  epoxy  quantity  and  die  placement 
pressure  must  be  controlled  around  some  target 
value  to  ensure  that  variation  in  the  epoxy  die  attach 
thickness  is  maintained  or  minimized  within  its  stated 
range.  This  is  accomplished  through  utilizing 
Statistical  Process  Control  (SPC).  SPC  was  used  to 
chart  the  performance  of  the  epoxy  weight  and  die 
placement  pressure  over  time.  This  not  only 
indicates  when  the  process  is  “out  of  control,"  but 
also  provides  valuable  information  which  can  be 
used  to  better  understand  the  variability  of  the 
process.  For  example,  if  the  die  placement  pressure 
changes  over  time,  then  utilizing  SPC  can  determine 
how  often  adjustments  of  the  pressure  apparatus  are 
necessary  to  maintain  or  minimize  variability  over 
time. 

7.5  PRODUCTION  AND  OPERATIONAL 
SUPPORT  PHASE  ACTIVITIES 

Production,  Operation,  and  Support  Requirements 
were  firmly  established  prior  to  the  start  of 
production.  The  customer  had  met  with  Ajax  to 
discuss  production  requirements,  such  as  process 
yield  targets,  statistical  process  control,  failure 
reporting,  analysis  and  corrective  action,  and  a 
continuous  measurable  improvement  (CMI)  program. 
The  QFD  was  again  updated  to  include  product  and 
process  changes  that  arose  after  EMD. 
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Ajax  was  famliar  wilh  many  of  the  success  stories  of 
companies  utilizing  continuous  measurable 
improvement  philosophies.  Ajax  had  adopted 
statistical  process  control,  variability  reduction,  and 
a  FRACAS  system  for  the  M51  and  M52  radar 
production  operations.  Ajax  was  currently 
experiencing  the  success  of  variability  reduction  in 
improved  process  yields.  The  plans  from  these  two 
programs  were  used  to  develop  the  plans  for  the 
current  contract.  The  plans  in  place  during  EMD 
required  only  minor  changes  prior  to  the  start  of 
production. 

The  FRACAS  plan  used  for  the  M50  and  M51  radar  DEFECT  ELIMINATION 

were  slightly  different  than  what  Ajax  had  used  in  the 

past.  In  the  past,  Ajax  tracked  failures  at  the  unit  and 

system  level.  The  new  system  would  include 

failures  occurring  at  module  level  test.  Ajax  hr  J 

found  that  many  of  the  problems  occurring  at  unit 

bum-in  could  have  been  detected  and  corrected 

during  module  level  build. 

In  addition,  Ajax  developed  a  field  failure  tracking 
system  to  track  the  performance  of  the  product  in  the 
field.  This  would  start  the  moment  the  product  left 
Ajax.  The  data  base  used  was  directly  tied  to  the 
FRACAS  system  so  part  and  failure  searches  could 
be  performed.  The  number  of  systems  to  be  tracked 
were  based  on  a  sample  of  the  total  population,  i.e., 
the  total  number  of  systems  built  for  the  customer  on 
this  contract.  The  system  documented  all  faults, 
identifying  cause  and  corrective  action.  This 
information  was  documented  as  lessons  learned  and 
used  to  identify  areas  of  improvement  for  the 
reliability  design  process.  Ajax  decision  to  use 
sampling  was  based  on  past  experience.  The  data 
received  on  past  contracts  were  incomplete.  The 
data  was  not  useful  for  drawing  definite  conclusions 
that  could  be  used  to  improve  the  reliability  design 
process.  This  was  attributed  to  the  difficulty 
involved  in  recording  detailed  failure  information 
and  maintaining  traceability.  This  was  also  a 
function  of  the  large  quantity  of  systems  produced 
ant  the  number  of  failures.  Ajax  felt  that  by  choosing 
a  manageable  number  of  samples  to  track  over  their 
life,  a  failure  history  record  could  be  developed  in 
which  critical  process  problems  could  be  identified, 
corrected,  and  used  as  lessons  learned  to  improve 
both  the  product  and  process. 
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8.0  INTRODUCTION 

This  handbook  describes  a  revised  process  for 
assuring  the  reliability  of  systems  and  their  hardware 
elements.  However,  modern  electronic  systems 
contain  increasing  dependance  on  computers  and 
their  software  for  system  control  and  execution  of 
functions.  Assuring  the  reliability  of  software  is 
therefore  crucial  to  the  reliability  of  electronic 
systems.  This  assurance  can  be  realized  by 
application  of  the  principles  and  approaches  of  this 
handbook  to  software  development. 

8.1  SUMMARY  OF  ACTIVITIES 

Each  of  the  acquisition  phases  are  described  in 
Chapters  two  through  six  in  terms  of  key  activities, 
each  having  inputs  and  outputs.  These  activities  are 
listed  as  follows: 

( 1 )  Quality  Evaluation 

(2)  Interpret  Customer  Needs 

(3)  Program  Planning 

(4)  Technology  Assessments  and  Testing 

(5)  Technical  Specifications 

(6)  Design  Reviews 

The  activity  designated  above  as  Technology 
Assessments  and  Testing"  is  a  composite 
representation  of  one  or  more  activities  within  each 
phase.  For  example,  the  EMD  phase  contains  four 
technology  assessments  and  test  activities  (5.1.4 
through  5.1.7)  which  are  summarized  here  as 
Technology  Assessments  and  Testing."  Each  of  the 
above  six  activities  is  applicable  to  software 
reliability. 

The  Quality  Evaluation  is  an  overall  assessment  of 
the  commitment  to  defect  free  hardware.  The 
software  analog  is  continuous  process  improvement 
in  accordance  with  the  Capability  Maturity  Model 
(CMM)  developed  by  the  Software  Engineering 
Institute  (SE1).  This  approach  provides  guidance  for 
maturing  and  measuring  the  software  development 
process  through  five  levels  of  achievement.  This 
process  is  defined  in  more  detail  in  paragraph  8.2. 
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Definition  of  customer  requirements  is  an  activity 
equally  applicable  to  both  hardware  and  software. 

Quality  Function  Deployment,  defined  as  a 
recommended  tool  for  interpreting  customer  needs, 
should  be  applied  to  the  development  of  software 
requirements  that  are  directly  traceable  to  customer 
needs.  This  activity  corresponds  to  the  System 
Requirements  Analysis/Design  and  Software 
Requirements  Analyses  activities  of  DOD-STD- 
2167A. 

Program  planning  is  as  essential  for  software  as  it  is 
for  hardware.  For  example,  this  activity  would  be 
used  to  develop  the  Software  Development  Plan 
defined  in  DOD-STD-21 67A.  Most  outputs  of  this 
activity,  as  described  in  Chapters  two  through  six, 
are  appropriate  to  software  reliability.  For  instance, 
these  outputs  include  CAE  Tools,  Technical 
Objectives  Flowdown  and  Benchmarking  and 
Training  Plans.  These  three  outputs  are  as 
appropriate  to  software  as  they  are  to  hardware. 

•  CAE  Tools  -  Specific  automated  software 
development  tools  must  be  defined  along  with 
all  required  hardware  and  training  for  the  use  of 
these  tools. 

•  Technical  Objectives  Flowdown  -  Software 
requirements,  including  quantitative  reliability 
requirements,  must  be  allocated  to  lower  levels 
in  the  software  hierarchy  in  a  manner 
comparable  to  the  flowdown  of  hardware 
requirement.  Traceability  from  lower  level 
requirements  back  to  customer  requirements 
must  be  maintained. 

•  Benchmarking  and  Training  Plans  -  The 
description  of  this  output  for  hardware  reliability 
is  directly  applicable  to  software  reliability. 
Software  benchmarking  can  be  done  both 
within  the  enterprise  and  external  to  the 
enterprise.  Its  purpose  is  to  employ 
dramatically  improved  processes  based  on 
"best  practices"  used  by  others.  Training  plans 
that  focus  on  deploying  "best  practices"  are  key 
to  developing  high  reliability  software.  These 
plans  should  cover  such  issues  as 
requirements  development,  coding  practices, 
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defect  reduction  techniques,  testing  methods 
and  software  configuration  management. 

The  software  equivalent  to  Technology  Assessment 
and  Testing”  consists  of  activities  required  to 
generate  detail  software  specifications,  interface 
documents,  coded  software,  quantitative  software 
defect  estimates,  software  test  procedures  and 
testing.  The  technology  issues  focus  primarily  on 
defect  prevention  actions  including  topics  such  as 
Lesson  Learned. 

The  technical  specifications  activity  consists  of  the 
development  of  the  deliverables  spelled  out  in  DOD- 
STD-2167A.  These  include  Software  Requirements 
Specifications,  Interface  Requirements 
Specifications,  Software  Design  Documents,  and 
Software  Specifications. 

Design  Reviews  are  a  critical  element  of  defect  SOFTWARE  REVIEWS 

prevention  programs.  Software  inspections  and 

walkthroughs  are  powerful  tools  for  improving  the 

quality  and  reliability  of  software.  These  software 

reviews  should  have  the  same  attributes  as  identified 

for  the  Design  Review  Activity  defined  in  Chapters 

two  through  six.  Primary  among  these  attributes  are 

issues  such  as  internal  consistency,  traceability  to 

customer  requirements,  and  documented 

procedures  for  conducting  both  internal  and  external 

reviews. 

Applying  a  quality  process  approach  to  software 
reliability  assurance  uses  the  methodology  defined 
in  Chapters  two  through  six.  As  applied  to  software 
this  consists  of  process  development  in  accordance 
with  the  SEI  maturity  model,  development  of  the 
products  described  in  DOD-STD-2167A, 
determination  of  expected  failure  rates  and  testing  in 
accordance  with  the  principles  outlined  in  the  6 
December  1991  draft  MIL-HDBK-XXX,  "Military 
Handbook,  Hardware/Software  Reliability  Assurance 
and  Control,"  and  the  implementation  of  a 
comprehensive  defect  prevention  program. 
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8.2  SOFTWARE  PROCESS  MATURITY 

Continuous  process  improvement  is  based  on  many  SOFTWARE  MATURITY 

small,  evolutionary  steps.  The  SEI  Capability 

Maturity  Model  (CMM)  provides  a  framework  for 

organizing  these  evolutionary  steps  into  five  maturity 

levels  that  lay  successive  foundations  for  continuous 

process  improvement.  These  five  maturity  levels 

define  scale  for  measuring  the  maturity  of  an 

organization's  software  process  and  lor  evaluating 

its  software  process  capability.  The  levels  also  help 

an  organization  prioritize  its  improvement  efforts. 

A  maturity  level  is  a  well-defined  plateau  toward 
achieving  a  mature  software  process.  Each  maturity 
level  provides  a  layer  in  the  foundation  for 
continuous  process  improvement.  Each  level 
comprises  a  set  of  process  goals  that,  when  satisfied, 
stabilize  an  important  component  of  the  software 
process. 

The  following  characterizations  of  the  five  maturity 
levels  highlight  the  primary  process  changes  made 
at  each  level: 

Level  1  -  The  Initial  Level 

At  the  Initial  Level,  the  organization  typically  does  not 
provide  a  stable  environment  for  developing  and 
maintaining  software.  When  an  organization  lacks 
sound  management  practices,  the  benefits  of  good 
software  engineering  practices  are  undermined  by 
ineffective  planning  and  reaction-driven  commitment 
systems. 

The  software  process  capability  of  Level  1 
organizations  is  unpredictable  because  the  software 
process  is  constantly  changed  or  modified  as  the 
work  progresses.  Schedules,  budgets,  functionality, 
and  product  quality  are  generally  unpredictable. 

There  are  few  stable  software  processes  in 
evidence,  and  performance  can  be  predicted  only  by 
individual  rather  than  organizational  capability. 
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Level  2  -  The  Repeatable  Level 

At  the  Repeatable  Level,  policies  for  managing  a 
software  project  and  procedures  to  implement  those 
policies  are  established.  Planning  ard  managing 
new  projects  is  based  on  experience  with  similar 
projects.  An  objective  in  achieving  Level  2  is  to 
institutionalize  effective  management  processes  for 
software  projects,  which  allows  organizations  to 
repeat  successful  practices  developed  on  earlier 
projects,  although  the  specific  processes 
implemented  by  the  projects  may  differ.  An  effective 
process  can  be  characterized  as  practiced, 
documented,  enforced,  trained,  measured,  and  able 
to  improve. 

Level  2  organizations  have  installed  basic  software 
management  controls.  Realistic  project 
commitments  are  based  on  the  results  observed  on 
previous  projects  and  on  the  requirements  of  the 
current  project.  The  software  managers  for  a  project 
track  software  costs,  schedules,  and  functionality, 
problems  in  meeting  commitments  are  identified 
when  they  arise.  Software  requirements  and  the 
work  products  developed  to  satisfy  them  are 
baselined,  and  their  integrity  is  controlled.  Software 
project  standards  are  defined,  and  the  organization 
ensures  they  are  faithfully  followed.  The  software 
project  works  with  its  subcontractors,  if  any,  to 
establish  a  strong  customer-supplier  relationship. 

Level  3  -  The  Defined  Level 

At  the  Defined  Level,  the  standard  process  for 
developing  and  maintaining  software  across  the 
organization  is  documented,  including  both  software 
engineering  and  management  processes,  and  these 
processes  are  integrated  into  a  coherent  whole.  This 
standard  process  is  referred  to  as  the  organization's 
standard  software  process.  Processes  established 
at  Level  3  are  used  (and  changed,  as  appropriate)  to 
help  the  software  managers  and  technical  staff 
perform  more  effectively.  The  organization  exploits 
effective  software  engineering  practices  when 
standardizing  its  software  processes.  There  is  a 
group  that  is  responsible  for  the  organization's 
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software  process  activities.  An  organization-wide 
training  program  is  implemented  to  ensure  that  the 
staff  and  managers  have  the  knowledge  and  skills 
required  to  fulfill  their  assigned  roles. 

A  well-defined  process  can  be  characterized  as 
including  readiness  criteria,  inputs,  standards  and 
procedures  for  performing  the  work,  verification 
mechanisms  (such  as  peer  reviews),  outputs,  and 
completion  criteria.  Because  the  software  process  is 
well  defined,  management  has  good  insight  into 
technical  progress  on  all  projects. 

Level  4  -  The  Managed  Level 

At  the  Managed  Level,  the  organization  sets 
quantitative  defect  goals  for  both  software  products 
and  process.  Productivity  and  defect  level  are 
measured  for  important  software  process  activities 
across  all  projects  as  part  of  an  organizational 
measurement  program.  An  organization-wide 
software  process  database  is  used  to  collect  and 
analyze  the  data  available  from  the  projects'  defined 
software  processes.  Software  processes  are 
instrumented  with  well-defined  and  consistent 
measurements  at  Level  4. 

These  measurements  establish  the  quantitative 
foundation  for  evaluating  the  project's  software 
processes  and  products. 

Projects  achieve  control  over  their  products  and 
processes  by  narrowing  the  variation  in  their  process 
performance  to  fall  within  acceptable  quantitative 
boundaries.  Meaningful  variations  in  process 
performance  can  be  distinguished  from  random 
variation  (noise),  particularly  within  established 
product  lines.  The  risks  involved  in  moving  up  the 
learning  curve  of  a  new  application  domain  are 
known  and  carefully  managed. 

Level  5  «  The  Optimizing  Level 

At  the  Optimizing  Level,  the  entire  organization  is 
focused  on  continuous  process  improvement.  The 
organization  has  the  means  to  identify  weaknesses 
and  strengthen  the  process  proactively,  with  the  goal 
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of  preventing  the  occurrence  of  delects.  Data  on  the 
effectiveness  of  the  software  process  is  used  to 
perform  cost  benefit  analyses  of  new  technologies 
and  proposed  changes  to  the  organization's 
software  process.  Innovations  that  exploit  the  best 
software  engineering  practices  are  identified  and 
transferred  throughout  the  organization. 

Software  project  teams  in  Level  5  organizations 
analyze  defects  to  determine  their  causes.  Software 
processes  are  evaluated  to  prevent  known  types  of 
defects  from  recurring,  and  lessons  learned  are 
disseminated  to  other  projects. 

The  software  process  capability  of  Level  5 
organizations  can  be  characterized  as  continuously 
improving  because  Level  5  organizations  are 
continuously  striving  to  improve  the  range  of  their 
process  capability,  thereby  improving  the  process 
performance  of  their  projects.  Improvement  occurs 
both  by  incremental  advancements  in  the  existing 
process  and  by  innovations  using  new  technologies 
and  methods. 

Each  of  the  maturity  levels  two  through  five  are 
characterized  by  the  availability  of  credible  metrics  to 
measure  planned  versus  actual  results.  These 
metrics  are: 

Level  Two: 

(1 )  Software  Size  -  This  metric  tracks  changes  in 
the  size  of  the  software  being  developed.  Size 
is  typically  specified  by  source  lines  of  code. 

(2)  Staffing  Profiles  -  This  tracks  the  project's 
ability  to  maintain  planned  staffing  levels  and 
sufficient  staffing  for  timely  completion  of  the 
program. 

(3)  Software  Units  Designed,  Tested  and 
Integrated. 

(4)  Computer  Resource  Utilization  (Throughput, 
Memory  and  Communications  Channels) 

(5)  Testing  of  Deliverables  -  This  metric  composes 
the  actual  versus  planned  system  testing 
results. 

(6)  Code  Errors  -  This  is  a  measurement  of  the  rate 
at  which  software  code  errors  are  identified  and 
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resolved. 

(7)  Cumulative  Code  Defect  Plot  -  This  metric 
measures  -cumulative  software  discrepancy 
reports  over  the  life  of  the  project. 

Level  Three: 

This  level  includes  the  metrics  described  for  level 

two  plus  the  following: 

(1 )  Design  Errors  -  This  measures  the  rate  of  which 
software  errors  are  identified  and  resolved 
during  the  design  phase  of  the  software 
development  process. 

(2)  Development  Progress  -  This  metric  monitors 
progress  by  combining  schedule  weighting 
factors,  percent  complete  estimates  and 
weighted  percent  complete  estimates  based  on 
relative  difficulty  of  development. 

Level  Four: 

Level  Four  metrics  contains  ievel  three  metrics  plus 

the  following: 

(1 )  Earned  Value  Index  -  This  a  composite  metric 
that  measures  work  progress  in  terms  of 
percentage  complete,  labor  hours  charged  and 
labor  hours  estimated. 

(2)  Production  Rate  -  This  metric  measures  the 
software  size  (lines  of  code)  per  total  labor  hour 
charged. 

(3)  Requirements  Traceability  -  Measurement  of 
requirements  changes  during  the  development 
cycle. 

(4)  Requirements  Coverage  -  This  is  the  ratio  of 
the  total  requirements  met  to  the  total  number 
of  requirements,  plotted  over  time. 

(5)  Design  Specification  Change  Rates  -  This  is 
the  measure  of  changes  caused  by  changed 
requirements  or  modified  design. 

Level  Five 

There  are  no  new  metrics  required  for  this  level. 
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The  SEI  maturity  methodology  is  expected  to  be 
used  throughout  all  acquisition  phases  with 
continuous  progress  towards  level  five  monitored 
through  the  appropriate  metrics  noted  above. 

8.3  SOFTWARE  FAILURE  RATES 

The  fundamental  focal  points  of  this  handbook  is  the 
prevention  and/or  elimination  of  failure.  Inherent  in 
this  focus  is  the  quantification  of  failure  rates.  Each 
of  the  acquisition  phase  descriptions  contains 
requirements  for  the  quantification  of  failure  rates 
both  through  an  allocation  process  in  the  early 
acquisition  phases  and  prediction  process  in  the 
later  acquisition  phases.  Similar  methodologies 
need  to  be  applied  to  software. 

Software  and  hardware  differ  in  several  respects.  HARDWARE  AND 

Software  does  not  wear  out.  Once  a  software  failure  SOFTWARE  FAILURES 

is  corrected  it  is  gone  forever;  many  hardware  faults 

can  recur.  However,  hardware  and  software 

reliability  are  very  similar.  Both  a  running  program 

and  an  operating  hardware  item  can  be  seen  as 

"black  boxes."  Every  once  in  a  while  the  black  box 

fails.  For  softwcre,  time  brings  with  it  a  succession  of 

input  states.  The  more  time  that  goes  by,  the  higher 

the  quantity  of,  and  the  more  variety  of,  input  states 

the  program  encounters.  Eventually,  because  of  the 

presence  of  defects,  an  input  state  will  trigger  a 

failure.  Thus  both  hardware  and  software  reliability 

can  be  modeled  as  random  processes. 

There  are  two  terms  relating  to  software  reliability 
that  are  used  in  the  following  paragraphs.  These 
terms  are  failure  and  defect.  As  used  herein,  a 
failure  is  an  observed  malfunction  of  the  software,  a 
defect  is  the  deficiency  that  causes  or  can  cause 
failure. 

A  software  failure  occurs  when  the  program 
produces  output  that  deviates  from  what  the 
requirements  specify.  A  failure  can  be  one  of 
conformance,  in  which  the  program  does  not 
produce  the  right  answer,  or  one  of  performance,  in 
which  the  program  does  not  perform  a  required 
function  in  a  timely  manner.  Performance  failures 
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indude  crashes,  hangs,  and  software  that  does  not 
meet  its  response  or  throughput  time  requirements. 

Software  failures  arise  from  a  population  of  software 
defects.  A  software  defect  is  missing,  extra,  or 
defective  code  that  has  caused  or  can  potentially 
cause  a  failure.  Every  time  a  defect  is  traversed 
during  execution,  a  failure  does  not  necessarily 
ensue;  it  depends  on  tie  machine  state  (values  of 
intermediate  variables).  The  extent  to  which  a 
program  contains  defects  may  be  expressed  in 
defects  per  thousand  lines  of  executable  source 
code  (KLOC). 

These  defects  can  occur  during  any  of  the  four 
primary  phases  of  software  development;  1 ) 
requirements  definition,  2)  preliminary  design,  3) 
detail  design  and  4)  coding.  Defects  during  these 
phases  are  affected  by  factors  such  as  the  frequency 
of  changes  to  program  specification,  programmer 
skill  and  the  volume  of  program  design 
documentation.  Control  of  these  factors  is  the  key  to 
failure  prevention. 

8.3.1  FAILURE  RATE  ALLOCATION 

Allocation  of  failure  rate  consists  of  finding  an 
achievable  combination  of  failure  rates  that  supports 
achievement  of  a  system's  or  subsystem’s 
requirements.  The  failure  rates  cannot  have  just  any 
values;  values  are  constrained  by  a  range  that  is 
realistically  achievable.  An  acceptable  allocation  is 
one  in  which  all  rates  lie  within  their  achievable 
ranges  and  the  overall  subsystem  or  system  meets 
or  exceeds  the  requirements.  The  rates  can  be 
chosen  arbitrarily  but  ideally  would  be  chosen  based 
on  intuition  and  on  experience  with  similar  and 
previous-generation  items.  If  the  allocated  values 
ensure  that  the  system  requirements  are  met,  the 
allocation  is  complete  and  any  "excess"  can  be 
either  allocated  to  another  part  of  the  system,  or 
reserved  as  a  means  of  mitigating  risk. 

The  failure  rate  allocation  process,  at  any  level  in  the 
software  hierarchy,  should  apply  a  "Six  Sigma" 
approach  to  software  defect  reduction  since  the 
prevention  of  defects  is  the  key  to  the  prevention  of 
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failures.  The  baseline  level  of  defects  is  based  on 
prior  experience.  It  is  recommended  that 
benchmarking  be  used  as  a  tool  to  identify  toe  best 
(i.e.  fewest  defects)  examples  of  prior  software 
developments.  Generally  this  benchmarking  can  be 
applied  within  an  organization.  The  objective  is  to 
identify  both  toe  lowest  defect  software  development 
and  the  processes  used  in  toe  development.  Having 
established  the  "best”  baseline,  toe  allocation 
process  must  establish  improvement  targets  to  be 
achieved  through  techniques  such  as  software 
simplification  and  check  lists  based  on  root  cause 
analysis. 

Once  the  system  functionality  is  partitioned  into 
hardware  and  software  subsystems,  toe  levels  of 
software  hierarchy  are  established  through  DOD- 
STD-2167A  methodology.  The  levels  of  software 
decomposition,  such  as  computer  software 
components  (CSCs)  and  computer  software 
configuration  items  (CSCIs),  refer  to  parts  of  the 
static  program  as  it  is  viewed  tor  the  purpose  of 
configuration  management.  When  executing,  the 
software  subsystem  will  exhibit  a  dynamic  structure. 
Allocation  should  take  place  at  the  level  at  which 
individual  threads  of  execution  {processes"  or 
"tasks")  exist.  Generally,  this  level  corresponds  to 
the  CSCI  level.  This  level  is  also  appropriate  for 
allocation  because  the  interfaces  among  modules 
are  included. 

-  The  allocation  techniques  identified  herein  are 

"Allocation  Based  on  Achievable  Failure  Rates," 

"Equal  Apportionment”,  "Proportional  Allocation," 
"Weighted  Allocation,"  "Constrained  Allocation,"  and 
"Re-allocation." 

The  bottom-up  allocation  method,  "Allocation  Based 
on  Achievable  Failure  Rates"  requires  the  ability  to 
estimate  CSCI  utilization  rates.  This  method 
provides  a  set  of  allocations  to  each  CSCI  which 
accurately  reflect  the  planned  usage  and  the 
execution  time  available  for  achieving  reliability 
growth.  If  failure  rate  allocations  provided  by  this 
method  do  not  support  achievement  of  the  system  or 
subsystem  requirements  it  is  an  indication  that  there 
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may  be  a  problem  with  the  specification 
requirements  or  one  or  more  higher  level  allocations. 

In  "Equal  Apportionment,"  the  components  of  an 
aggregate  are  allocated  equal  rates  in  such  a  way 
that  the  aggregate  meets  its  reliability  goal. 

"Proportional  Allocation"  takes  into  account  the 
length  of  time  each  component  is  active,  the  longer 
a  component  is  active,  the  more  exposure  it  has  to 
the  possibility  of  failure.  The  premise  of  proportional 
allocation  is  that  higher  reliability  should  be 
demanded  of  components  that  active  for  a  greater 
share  of  the  time  relative  to  the  other  components. 

In  "Weighted  Allocation,"  the  rate  allocated  to  a 
component  is  based  on  the  criticality  of  the 
component  and/or  feasibility  of  its  meeting  a 
reliability  objective.  The  criticality  of  the  component 
includes  the  consequences  of  the  failure  to  mission 
success  and  safety. 

In  "Constrained  Allocation,"  the  allocation  is 
optimized  with  respect  to  additional  considerations 
such  as  cost. 

In  "Re-allocation,"  a  previous  allocation  is  revised 
because  one  or  more  components  could  not  meet 
their  reliability  objectives. 

Details  concerning  these  allocations  techniques  can 
be  found  in  Rome  Laboratory  Report  RL-TR-92-15, 
"Reliability  Techniques  for  Combined  Hardware  and 
Software  Systems,"  or  MIL-HDBK-XXX,  Draft  Military 
handbook,  "Hardware/Software  Reliability 
Assurance  and  Control,"  dated  6  December  1 991 . 

8.3.2  FAILURE  RATE  PREDICTIONS 

Hardware  reliability  prediction  provides  a  failure  rate 
which,  in  theory,  is  the  best  reliability  achievable  as 
the  development  and  manufacturing  processes  are 
perfected.  Software  does  not  have  the  same  kind  of 
limitation.  The  reliability  of  software  will  generally 
improve  over  time  as  failures  are  uncovered  through 
testing  and  are  fixed.  When  software  reliability  is 
predicted,  that  prediction  must  be  related  to  a  point  in 
time. 
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The  technique  for  software  failure  rate  prediction 
uses  metrics  derived  from  characteristics  of  the 
software  development  process  and  of  the  products. 
They  are  consistent  with  the  metrics  required  as  part 
of  the  SEI  Maturity  Model  approach  to  continuous 
improvement.  These  metrics  are  as  follows: 

Metrical  -  is  the  number  of  errors  in  the  Software 
Requirements  Specification  (SRS). 

Metric  #2  -  is  the  number  of  requirements  statements 
in  the  SRS. 

Metric  #3  -  is  the  number  of  pages  in  the  SRS,. 
Writing  style  will  account  for  a  small,  unavoidable, 
variation  in  this  metric. 

Metric  #4  -  is  the  effort  expended,  in  man-months,  on 
the  requirements  analysis  phase. 

Metric  #5  -  is  the  number  of  changes  (corrections 
and  modifications)  to  the  SRS  after  it  has  been  place 
under  configuration  control. 

MetricJS  -  is  the  number  of  errors  in  preliminary 
design  documents. 

MstrteJfZ  -  is  the  number  of  Computer  Software 
Components  (CSCs)  in  the  software  structure. 

Metric  #8  -  is  the  number  of  Computer  Software  Units 
(CSUs)  in  the  design  structure. 

Metric  #9  -  is  the  number  of  pages  in  the  Software 
Design  Documents  (SDDs). 

Metric  #10  -  is  the  number  of  man-months  expended 
for  preliminary  design. 

Metric  #1 1  -  is  the  average  number  of  times  a  unit  is 
tested  by  the  programmer  during  CSU  testing. 

Metric  #12  -  is  the  sum  of  the  number  of  design 
errors  identified  after  the  SDD  has  been  placed 
under  configuration  control,  the  number  of  design 
errors  identified  as  the  result  of  internal  reviews,  and 
the  number  of  faults  found  through  code  reviews  and 
related  inspections. 
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MetricJi3  •  is  the  total  number  of  executable  lines  of 
source  code. 

As  noted,  these  process  metrics,  are  consistent  with 
the  SEI  maturity  index  metrics  described  in 
paragraph  8.3.1 .  When  used  in  conjunction  with 
experience  based  formulae,  the  above  data  permit 
the  assessment  of  initial  software  failure  rates  at 
various  stages  of  development.  These  stages  are 
requirements  analysis,  preliminary  design,  detail 
design,  coding,  and  software  module/system  testing. 

The  empirical  formulae  may  be  found  in  Section  5  of 
MIL-HDBK-XXX,  Draft  Military  Handbook: 
"Hardware/Software  Reliability  Assurance  and 
Control,"  dated  6  December  1991 .  This  draft 
handbook  also  contains  a  procedure  for  predicting 
failure  rates  very  early  in  the  development  process 
when  only  the  program  size  and  processor  speed 
may  be  known. 

The  flowdown  of  quantitative  failure  rate 
requirements  and  the  prediction  of  achieved  values 
as  detail  knowledge  becomes  available,  sets  the 
stage  for  preventing/eliminating  failures  through  the 
use  of  software  reviews  coupled  with  root  cause 
analysis. 

8.4  DEFECT  PREVENTION  TECHNIQUES 

To  achieve  reliable  software  products,  most 
development  processes  rely  on  defect  detection  and 
correction  through  inspections,  walkthroughs,  and 
reviews  early  in  the  development  cycle,  and  through 
extensive  testing. 

Inspections  and  walkthroughs  are  peer/customer 
examinations  aimed  at  assisting  the  program 
developers  in  improving  their  work  by  detecting 
defects.  They  are  the  direct  analog  of  the  Design 
Review  Activities  described  in  Chapters  two  through 
six. 

Software  inspection  is  a  powerful  technique  for 
improving  the  quality  of  software  development.  This 
objective  is  achieved  by  helping  the  software 
developer  recognize  and  fix  their  own  errors  early  in 
the  process  and  to  gather  data  on  defects  that  can  be 
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used  for  improving  the  software  development 
process,  inspections  are  labor  intensive  but  are  very 
effective  for  eliminating  defects.  The  inspection 
process  involves  a  peer  review  of  the  developing 
software  with  emphasis  on  error  detection  and 
correction.  These  inspections  should  be  conducted 
at  every  point  in  the  development  process.  In 
addition  to  inspecting  new  product  elements,  every 
change  should  be  inspected  and  re-inspections  of 
an  entire  element  are  needed  when  there  is 
substantial  change  activity  or  if  inspection/test  resuits 
indicate  unusual  problems.  Checklists  are  also 
important  to  the  success  of  the  inspection,  insuring 
that  all  details  of  the  process  are  covered.  In  all 
cases,  the  inspection  should  take  place  with  a  clear 
set  of  entry  and  exit  criteria  in  place. 

As  defects  are  identified  they  are  recorded  and 
discussed.  Particular  attention  is  paid  to  previously 
undetected  defects.  The  developer  of  the  software 
product  is  charged  with  the  responsibility  tor 
correcting  the  error.  The  corrected  products  must  be 
re-submitted  to  the  inspection  team  to  verify  the 
adequacy  of  the  correction.  A  complete  inspection 
process  will  consist  of  planning,  preliminary  review, 
inspection  meeting,  defect  correction  and  follow-up. 

A  walkthrough  is  a  less  intensive  review  of  a 
software  product  than  an  inspection.  They  are  also 
less  formal  and  do  not  generally  involve  the  rework 
and  follow-up  required  in  inspections.  Errors  found 
during  a  walkthrough  are,  however,  documented, 
usually  in  the  form  of  a  results  report. 

True  defect  prevention  requires  combining  causal 
analysis  with  inspections/walkthroughs.  Defects  are 
analyzed  to  determine  the  cause  of  the  error,  how  to 
prevent  the  error  and  how  to  remove  similar  defects 
that  may  exist  in  the  rest  of  the  software.  This  causal 
analysis  should  be  done  by  the  software 
development  team  during  the  development  process. 
Causal  analysis  by  the  developer  making  the  error 
results  in  a  more  accurate  determination  of  the  true 
error  cause  and  more  relevant  prevention  actions. 
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There  are  four  key  elements  in  a  defect  prevention 
process:  1}  systematic  causal  analysis,  2) 
management  support  3)  on-going  development 
team  meetings  and  4)  a  database  and  tools  for  data 
collection  and  tracking  of  actions. 

The  causal  analysis  should  take  place  once  the 
defects  from  a  software  development  stage  have 
been  discovered  and  corrected  by  virtue  of 
inspections  and  walkthroughs.  For  each  defect  the 
following  questions  can  be  posed: 

1 )  What  is  the  error  category  (communication, 
oversight,  education  or  transcription) 

2)  How  was  the  error  introduced  or  caused 

3)  At  what  stage  was  the  error  created 

4)  How  can  the  error  be  prevented  in  the  future 
and  how  can  similar  errors  be  detected  and 
removed  from  other  elements  of  the  program. 

Repeated  causal  analyses  reviews  should  take 
place  during  any  software  development  phases 
during  which  numerous  errors  are  likely  to  be 
detected. 

Preventative  actions  resulting  from  these  causal 
analysis  reviews  generally  fall  into  several 
categories: 

1)  Process  improvements 

2)  Tool  enhancement 

3)  Education 

4)  Software  product  changes 

5)  Communications  improvements 

These  preventative  actions  are  reported  and  saved 
in  a  lessons  learned  file  that  may  include  items  such 
as  error  lists,  inspection  or  walkthrough  checklists, 
coding  or  performance  guidelines,  and  training 
improvements. 

8.5  CONCLUSIONS 

The  approach  to  software  defect  prevention 
described  in  this  chapter,  parallels  the  hardware 
process  described  in  Chapters  2  through  6.  A  frame 
work  for  ensuring  the  implementation  of  continuous 
improvement  during  all  acquisition  phases  is 
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provided  by  the  SEI  Capability  Maturity  Model.  Each 
acquisition  phase  requires  many  of  the  same 
activities  and  outputs  for  both  hardware  and 
software.  Quantitative  measures  of  defect  levels  for 
software  can  be  allocated  and  predicted  in  a  manner 
similar  to  that  applied  to  hardware.  The 
development  of  these  defect  estimates  use  many  of 
the  metrics  required  by  the  SEI  model.  Defect 
prevention  takes  place  through  the  continuous 
application  of  design  reviews  and  the 
implementation  of  a  systematic  causal  analysis 
process. 
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Rome  Laboratory  plans  and  executes  an  interdisciplinary 
program  in  research,  development,  test,  and  technology 
transition  in  support  of  Air  Force  Command,  Control, 
Communications  and  Intelligence  (C3I)  activities  for  all 
Air  Force  platforms.  It  also  executes  selected 
acquisition  programs  in  several  areas  of  expertise. 
Technical  and  engineering  support  within  areas  of 
competence  is  provided  to  ESC  Program  Offices  (POs)  and 
other  ESC  elements  to  perform  effective  acquisition  of 
C3I  systems.  In  addition,  Rome  Laboratory's  technology 
supports  other  AFMC  Product  Divisions,  the  Air  Force  user 
community,  and  other  DOD  and  nor. -OOD  agencies.  Rome 
Laboratory  maintains  technical  competence  and  research 
programs  in  areas  including,  but  not  limited  to, 
communications,  command  and  control,  battle  management, 
intelligence  information  processing,  computational 
sciences  and  software  producibility,  wide  area 
surveillance/sensors,  signal  processing,  solid  state 
sciences,  photonics,  electromagnetic  technology, 
superconductivity,  and  electronic 
reliability/maintainability  and  testability. 


